Frank
8f129180d8
Staged 59 GiB of retired LLM payloads on Dankclaw to ~dank/.AGENTTRASH/20261002-retired-llm-payloads/ (vLLM MiMo BF16, Qwen3.5-9B MTP source, both DFlash dirs, Gemma4-12B-QAT, SGLang MiMo INT8) and corrected every doc claim about what is on disk: gufo's rollback pair (Q4_K_XL weights + the Q8_K_XL DFlash2 drafter) and frank-small's Q6_K stay in place, so the rollback paths in the tags' env files still resolve. The vLLM-era frank-large fetch trio moves to old/ with its runtime; runbook links updated.
187 lines
9.7 KiB
Markdown
187 lines
9.7 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
|
||
description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
|
||
tags:
|
||
- llm
|
||
- frank-large
|
||
- dankclaw
|
||
- vllm
|
||
- strix-halo
|
||
- mimo
|
||
- mtp
|
||
- spec-decode
|
||
- bf16
|
||
timestamp: 2026-10-01
|
||
---
|
||
|
||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||
|
||
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
|
||
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
|
||
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
|
||
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
|
||
> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
|
||
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
|
||
|
||
## Identity
|
||
|
||
- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
|
||
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
|
||
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
|
||
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
|
||
`podman run` driven by `scripts/serve-frank-large.sh` under a root unit
|
||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
|
||
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
|
||
(17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8
|
||
W8A16 AutoRound conversion on the 3090, here unquantized.
|
||
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
|
||
are loaded into the draft pass).
|
||
|
||
## Why this shape (the roof math)
|
||
|
||
Decode speed for a dense model on Strix Halo is bandwidth-bound:
|
||
BF16 weights x one pass per token against the LPDDR5x aperture. Measured
|
||
2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
|
||
a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
|
||
because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
|
||
wall is generating more than one token per weight pass: **speculative decode
|
||
is the performance feature of this endpoint**, not a garnish.
|
||
|
||
The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
|
||
attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
|
||
the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
|
||
sequence. `--max-model-len 262144` (the native window) with
|
||
`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
|
||
four-session contract the gufo shape ran, with the pool shared by the
|
||
scheduler instead of carved per session.
|
||
|
||
## Spec method: cross-model MTP — measured, not assumed
|
||
|
||
| arm | prefill | decode | acceptance |
|
||
|---|---|---|---|
|
||
| BF16 base | 17,876 t/s | 13.1 t/s | — |
|
||
| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
|
||
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
|
||
|
||
One session, temperature 0, 28,647-token prefill / 512-token decode, same
|
||
client for all three arms (2026-10-01 bench window, frank-large stopped per
|
||
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
|
||
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
|
||
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
|
||
`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
|
||
fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
|
||
the compute queues) — this endpoint's thinking-heavy traffic spends its wall
|
||
clock in decode, which is the regime where that trade wins.
|
||
|
||
**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
|
||
Qwen3.5-9B that preserves the hidden-state geometry — identical
|
||
4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
|
||
per-position acceptance is the direct measurement that the distillation
|
||
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
|
||
(verified in both the BF16 base and the INT8 conversion) — which is exactly
|
||
why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
|
||
reads the head from the TARGET checkpoint, vLLM reads the draft from a
|
||
separate path. That difference is the whole trick:
|
||
|
||
```
|
||
--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
|
||
```
|
||
|
||
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
|
||
[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint
|
||
`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
|
||
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
|
||
|
||
## The numerics trap (greedy text is NOT preserved by spec decode)
|
||
|
||
Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
|
||
`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
|
||
reduction order, argmax near-ties diverge, and the divergence cascades.
|
||
vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
|
||
The gufo-era "greedy == autoregressive text" law does not carry over; do not
|
||
chase text drift between spec and no-spec arms as a regression.
|
||
|
||
## Payload
|
||
|
||
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
|
||
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
|
||
byte-count + LFS sha256-verified by
|
||
`old/download-frank-large-mimo.sh` (`--verify-only` re-checks; no token on
|
||
the curl command line).
|
||
- MTP source: `old/download-frank-large-qwen35.sh` →
|
||
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
|
||
`model.safetensors.index.json`; the shards carry the odd HF asset names
|
||
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
|
||
the engine boots from exactly this tree).
|
||
- DFlash drafter: `old/download-frank-large-dflash.sh` →
|
||
`/home/dank/models/Qwen3.5-9B-DFlash/`.
|
||
- **Payload status 2026-10-02:** all three trees moved to
|
||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
|
||
the other retired frank payloads) and the fetch trio moved to `old/` — the
|
||
endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
|
||
- The serve script gates the image by REPO DIGEST (rootless podman resolves
|
||
tags from the network; the pin exists to boot exactly these bytes
|
||
offline), then config + index + exact shard count, then the spec draft
|
||
(either weight layout: single `model.safetensors` or a sharded index),
|
||
then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
|
||
Dankserver 2026-09-29 lesson).
|
||
|
||
## Client contract
|
||
|
||
- **Alias:** `frank-large` (the `id` at `/v1/models`).
|
||
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
|
||
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
|
||
`models.yml` compat block for this provider sets
|
||
`reasoningContentField: "reasoning"`. Miss that and thinking silently
|
||
vanishes from transcripts.
|
||
- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
|
||
- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
|
||
serve script passes `--language-model-only` — the endpoint stays
|
||
text-only by decision; dropping the flag brings vision back.
|
||
- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
|
||
(temp 0.7, top_p 0.8) via `compat.extraBody`, and
|
||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
|
||
`chat_template_kwargs` form is what vLLM reads; a top-level
|
||
`preserve_thinking` is ignored there.
|
||
- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP
|
||
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
|
||
the env file with `set -a`.
|
||
|
||
## Ops
|
||
|
||
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
|
||
- The serve script sweeps stale container identities before starting:
|
||
`frank-large-mimo` itself and the bench-window container `vllm-mimo`.
|
||
- Container death → `podman wait` returns non-zero from ExecStart →
|
||
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
|
||
frank-primary lesson).
|
||
- Bench windows: `systemctl stop frank-large` is enough (mask fails on the
|
||
relinked unit; the deploy restores).
|
||
- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
|
||
boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
|
||
262,144 tokens per request: 3.13x` — four sessions share the pool
|
||
(the scheduler, not a carve). Live battery green: alias `frank-large`,
|
||
reasoning split into `reasoning`, tool_calls with correct args, and
|
||
`/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
|
||
matching the bench window.
|
||
|
||
## Rollback
|
||
|
||
Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
|
||
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
|
||
payload + DFlash2 drafter, all still on the host. Restoring the tag's
|
||
unit/env/serve trio and redeploying returns the 27B endpoint.
|
||
|
||
# Citations
|
||
|
||
- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
|
||
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
|
||
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
|
||
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
|
||
- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
|
||
- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch:
|
||
[[frank-small-sglang-mimo-v26]]
|