frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
Frank committed
1 parent
7da504a651
commit
0c5e054fb8
12 files changed
+1005
-328
No files matched your search
@@ -0,0 +1,156 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
|
||||
description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- sglang
|
||||
- rtx3090
|
||||
- qwen
|
||||
- mtp
|
||||
- vision
|
||||
- int8
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
|
||||
> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
|
||||
> (7771), same alias (`frank-small`), same client contract surface.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||||
systemd unit `scripts/frank-small.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
|
||||
tunables from `scripts/frank-small.env`.
|
||||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||||
|
||||
## Shape
|
||||
|
||||
- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
|
||||
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
|
||||
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
|
||||
tower (27 layers) in the safetensors — no separate mmproj file.
|
||||
- **Context:** four 151,552-token slots (`--context-length 151552`,
|
||||
`--max-running-requests 4`) against a **pinned** 606,208-token pool
|
||||
(`--max-total-tokens`). A single request may grow to the whole pool —
|
||||
a 170,019-token prompt is verified — the grid is the four-way
|
||||
concurrency contract.
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
ssm state pool is a first-class consumer. The engine autofills the
|
||||
mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
|
||||
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
|
||||
MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
|
||||
leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
|
||||
CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
|
||||
dies in capture, or later in the first long request once Triton's
|
||||
deferred kernel loads eat the remainder). Hence
|
||||
`--disable-prefill-cuda-graph`; decode graphs are untouched.
|
||||
- **Quantization:** `--quantization auto-round` — the checkpoint's
|
||||
`quantization_config.json` declares `quant_method: "auto-round"` and
|
||||
SGLang loads it natively (vision tower, `embed_tokens` and
|
||||
`linear_attn.in_proj` stay fp16).
|
||||
|
||||
## Spec method
|
||||
|
||||
**None — and this is a checkpoint fact, not a tuning choice.** The
|
||||
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
|
||||
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
|
||||
`model.safetensors.index.json` for both this INT8 conversion and the base
|
||||
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
|
||||
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
|
||||
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
|
||||
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
|
||||
paying the 4-token verify tax for no token. The serve script passes no
|
||||
`--speculative-*` flag, and the seam gates that.
|
||||
|
||||
## CUDA 13 toolchain (the sharp edge)
|
||||
|
||||
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
|
||||
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
|
||||
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
|
||||
`--compress-mode`) and its headers clash with the 13.0 device passes.
|
||||
|
||||
Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
|
||||
|
||||
| Package | Supplies |
|
||||
|---|---|
|
||||
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
|
||||
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
|
||||
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
|
||||
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
|
||||
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
|
||||
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
|
||||
|
||||
Three local modifications the tree carries:
|
||||
|
||||
1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
|
||||
`-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
|
||||
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
|
||||
`../../../include` / `../../../lib64` (the deb ships them as real
|
||||
near-empty dirs; the stub pass must see the 13.0 headers, not
|
||||
`/usr/include` 12.4).
|
||||
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
|
||||
includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
|
||||
`noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
|
||||
the unpatched 13.0 declarations are incompatible in the C++ host stub).
|
||||
|
||||
JIT caches: `~/.cache/sglang/jit/sm86/` and
|
||||
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
|
||||
change lands.
|
||||
|
||||
## Payload
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
|
||||
re-checks). The serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-small` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||||
as llama.cpp; models.yml sets `reasoningContentField:
|
||||
"reasoning_content"`). `--reasoning-parser qwen3`.
|
||||
- **Tools:** `--tool-call-parser qwen3_coder`.
|
||||
- **Vision:** supported (in-checkpoint tower).
|
||||
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
|
||||
`--top-k` — sampling rides on the request; the harness sends the Qwen
|
||||
recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
|
||||
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
|
||||
has no effort gate).
|
||||
|
||||
## Measured
|
||||
|
||||
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
|
||||
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
|
||||
above is what makes that compile succeed).
|
||||
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
|
||||
- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
|
||||
- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of
|
||||
live KV at once).
|
||||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
|
||||
remain intact on disk; its unit/env/serve trio is in `old/` after the
|
||||
cutover.
|
||||
|
||||
# Citations
|
||||
|
||||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
- SGLang: https://github.com/sgl-project/sglang
|
||||
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
|
||||
Reference in new issue
Block a user