Frank
dab483c79f
Engine: digest-pinned docker.io/kyuz0/vllm-therock-gfx1151 (vLLM 0.30, TheRock ROCm 10, Triton GDN kernels); the serve script gates the image by REPO DIGEST, the payload (config + index + exact shard count), the spec draft (BOTH weight layouts: single model.safetensors for dflash, sharded index for the Qwen3.5-9B MTP head), and whole /dev/dri, sweeps the bench identity vllm-mimo, then exec podman wait so a container death trips Restart=always. Shape: 4 x 262144 at util 0.45 (KV grows only on the 8 full-attention layers), MTP 2 draft tokens. Measured 2026-10-01 window (frank-large stopped): base 13.1 t/s decode (~94% of the LPDDR5x BF16 roof), DFlash-8 17.4, MTP-2 19.8 with 71%/44% per-position acceptance - the cross-model head works because vLLM reads the draft from a separate path while MiMo ships zero mtp.* tensors. Spec arms do not preserve greedy bf16 text (batched-verify reduction order). The seam was rewritten around the new laws (18 tests): unit self-healing, digest gating, engine wiring, the measured spec table, the retired gufo gates, the deployer MainPID check. frank_small port test follows the PORT -> FRANK_LARGE_PUBLISH rename. Rollback: tag frank-large-gufo-q4kxl-20261001.
98 lines
5.4 KiB
Bash
98 lines
5.4 KiB
Bash
# frank-large.env — vLLM production config for frank-large (Dankclaw iGPU :7770).
|
|
#
|
|
# 2026-10-01 cutover from gufo (git tag frank-large-gufo-q4kxl-20261001).
|
|
# Target: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 precision
|
|
# (17.53 GiB, 4 shards) — the same architecture frank-small serves as an
|
|
# INT8 W8A16 AutoRound conversion on the 3090, here unquantized. The engine
|
|
# is the kyuz0 TheRock vLLM build, the only fleet-carried runtime path for
|
|
# gfx1151 (official vLLM images are CUDA-only).
|
|
#
|
|
# MEASURED 2026-10-01 bench window on this host (identical client, one
|
|
# session, temperature 0, 28,647-token prefill / 512-token decode):
|
|
#
|
|
# arm prefill decode acceptance (per position)
|
|
# BF16 base 17,876 t/s 13.1 t/s —
|
|
# BF16 + DFlash 8 tok 7,976 t/s 17.4 t/s pos0 59% pos1 26%; 14.8% token rate
|
|
# BF16 + MTP 2 tok 7,380 t/s 19.8 t/s pos0 71% pos1 44%; 57.4% token rate
|
|
#
|
|
# Decode laws: 13.1 t/s is ~94% of the Strix Halo LPDDR5x roof for BF16
|
|
# weights (17.5 GiB/token against ~256 GB/s) — a quant, not a bug; the same
|
|
# INT8 payload decodes 83.9 t/s on the 3090 because it moves ~12 GiB
|
|
# through ~936 GB/s. Speculative arms are the ONLY way past the wall, and
|
|
# both drafters pay ~55-59% of prefill (draft passes share the compute
|
|
# queues). Frank's call enabled MTP: the thinking-heavy traffic on this
|
|
# endpoint spends its wall clock in decode.
|
|
#
|
|
# MTP IS CROSS-MODEL: the head comes from Qwen/Qwen3.5-9B (the distill's
|
|
# source model, 15 mtp.* tensors, identical 4096-hidden/248320-vocab
|
|
# geometry) — MiMo itself ships ZERO mtp tensors, which is why frank-small's
|
|
# SGLang NEXTN is a 0.00-acceptance ghost (SGLang reads the head from the
|
|
# TARGET checkpoint; vLLM reads the draft from a separate path). The 71%/44%
|
|
# per-position acceptance is the distillation preserving Qwen3.5 hidden-state
|
|
# structure; it is measured, not assumed.
|
|
#
|
|
# NUMERICS NOTE (measured): with a spec drafter enabled, temperature-0 text
|
|
# DIFFERS from the no-spec run (greedy sha 1318aeb6 base vs 0208b45d dflash
|
|
# vs 20c41c38 mtp): batched verification changes the bf16 reduction order,
|
|
# so argmax near-ties diverge and cascade. Spec decode on vLLM is
|
|
# distribution-preserving, not bitwise-greedy-preserving. The gufo
|
|
# greedy==AR law does not carry over.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# FRANK_LARGE_-prefixed so none can leak.
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned (the :latest tag of this fork moves with no notice):
|
|
# image ID 2d24a8c5a3fd, vLLM 0.30.0 / torch 2.11.0 / ROCm 10.0.0 (TheRock),
|
|
# verified to ship vllm/v1/spec_decode/dflash.py, the qwen3_5 model registry
|
|
# entry (Qwen3_5ForConditionalGeneration), and the Triton GDN backend.
|
|
FRANK_LARGE_IMAGE=docker.io/kyuz0/vllm-therock-gfx1151@sha256:98c1f3ac9833e7fa45e9a82fb7a4315e6d95daaf2548a211c8da3a871b17533a
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_LARGE_MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16
|
|
FRANK_LARGE_MODEL_SHARDS=4
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# Same port and served alias as every frank-large predecessor; the harness
|
|
# models.yml id is unchanged. The bench-window container (vllm-mimo) is
|
|
# swept before every start.
|
|
FRANK_LARGE_CONTAINER=frank-large-mimo
|
|
FRANK_LARGE_LEGACY_CONTAINERS="vllm-mimo"
|
|
FRANK_LARGE_SERVED_NAME=frank-large
|
|
FRANK_LARGE_PUBLISH=7770:7770
|
|
|
|
# --- engine shape (2026-10-01 window) ----------------------------------------
|
|
# Full native window 262,144 at UTIL 0.45 booted clean on gfx1151 in all
|
|
# three arms (~200-240 s warm start). KV grows only on the 8 full-attention
|
|
# layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B); the 24 GDN
|
|
# layers keep a FIXED ~25 MiB state per running sequence, so the pool is
|
|
# generous and sessions are cheap. SEQS 4 preserves the four-session
|
|
# contract the gufo shape ran; vLLM shares the pool across them by
|
|
# scheduler, unlike gufo's per-session carve.
|
|
FRANK_LARGE_MAXLEN=262144
|
|
FRANK_LARGE_SEQS=4
|
|
FRANK_LARGE_UTIL=0.45
|
|
|
|
# --- speculative decode: cross-model MTP (measured +51% decode) ---------------
|
|
# mtp beats dflash here (19.8 vs 17.4 t/s) and needs 2 draft tokens, not 8:
|
|
# acceptance past position 1 is 44% and falling (DFlash's own curve decays
|
|
# 59/26/14/7). num_speculative_tokens beyond 2 is UNMEASURED — do not raise
|
|
# it without a fresh A/B. To flip drafters, set FRANK_LARGE_SPEC=dflash and
|
|
# repoint SPEC_DIR (the DFlash drafter stays on disk).
|
|
# The gufo-era law "spec decode is absent" was a frank-primary finding; this
|
|
# endpoint's traffic is decode-dominated thinking, which is the regime where
|
|
# spec wins (prefill costs ~55% in every arm).
|
|
FRANK_LARGE_SPEC=mtp
|
|
FRANK_LARGE_SPEC_DIR=/home/dank/models/Qwen3.5-9B
|
|
FRANK_LARGE_SPEC_TOKENS=2
|
|
|
|
# --- client contract ----------------------------------------------------------
|
|
# vLLM emits the thinking text in `reasoning` (NOT llama.cpp/gufo's
|
|
# `reasoning_content`) — the harness models.yml compat block for this
|
|
# provider must set reasoningContentField accordingly. The qwen3 parser is
|
|
# what keeps thinking out of `content`. Tools ride the qwen3_coder parser
|
|
# with auto tool choice. --language-model-only drops the vision tower (the
|
|
# endpoint stays text-only; the tower can come back by dropping the flag).
|
|
FRANK_LARGE_REASONING_PARSER=qwen3
|
|
FRANK_LARGE_TOOL_PARSER=qwen3_coder
|