Frank
0c5e054fb8
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
157 lines
7.4 KiB
Markdown
157 lines
7.4 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
|
||
description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
|
||
tags:
|
||
- llm
|
||
- frank-small
|
||
- dankclaw
|
||
- sglang
|
||
- rtx3090
|
||
- qwen
|
||
- mtp
|
||
- vision
|
||
- int8
|
||
timestamp: 2026-10-01
|
||
---
|
||
|
||
# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||
|
||
> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
|
||
> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
|
||
> (7771), same alias (`frank-small`), same client contract surface.
|
||
|
||
## Identity
|
||
|
||
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
|
||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||
systemd unit `scripts/frank-small.service` (`User=dank`,
|
||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
|
||
tunables from `scripts/frank-small.env`.
|
||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||
|
||
## Shape
|
||
|
||
- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
|
||
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
|
||
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
|
||
tower (27 layers) in the safetensors — no separate mmproj file.
|
||
- **Context:** four 151,552-token slots (`--context-length 151552`,
|
||
`--max-running-requests 4`) against a **pinned** 606,208-token pool
|
||
(`--max-total-tokens`). A single request may grow to the whole pool —
|
||
a 170,019-token prompt is verified — the grid is the four-way
|
||
concurrency contract.
|
||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||
dim x 1 B), pool 9.31 GiB.
|
||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||
ssm state pool is a first-class consumer. The engine autofills the
|
||
mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
|
||
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
|
||
MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
|
||
leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
|
||
CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
|
||
dies in capture, or later in the first long request once Triton's
|
||
deferred kernel loads eat the remainder). Hence
|
||
`--disable-prefill-cuda-graph`; decode graphs are untouched.
|
||
- **Quantization:** `--quantization auto-round` — the checkpoint's
|
||
`quantization_config.json` declares `quant_method: "auto-round"` and
|
||
SGLang loads it natively (vision tower, `embed_tokens` and
|
||
`linear_attn.in_proj` stay fp16).
|
||
|
||
## Spec method
|
||
|
||
**None — and this is a checkpoint fact, not a tuning choice.** The
|
||
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
|
||
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
|
||
`model.safetensors.index.json` for both this INT8 conversion and the base
|
||
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
|
||
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
|
||
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
|
||
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
|
||
paying the 4-token verify tax for no token. The serve script passes no
|
||
`--speculative-*` flag, and the seam gates that.
|
||
|
||
## CUDA 13 toolchain (the sharp edge)
|
||
|
||
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
|
||
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
|
||
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
|
||
`--compress-mode`) and its headers clash with the 13.0 device passes.
|
||
|
||
Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
|
||
|
||
| Package | Supplies |
|
||
|---|---|
|
||
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
|
||
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
|
||
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
|
||
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
|
||
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
|
||
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
|
||
|
||
Three local modifications the tree carries:
|
||
|
||
1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
|
||
`-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
|
||
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
|
||
`../../../include` / `../../../lib64` (the deb ships them as real
|
||
near-empty dirs; the stub pass must see the 13.0 headers, not
|
||
`/usr/include` 12.4).
|
||
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
|
||
includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
|
||
`noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
|
||
the unpatched 13.0 declarations are incompatible in the C++ host stub).
|
||
|
||
JIT caches: `~/.cache/sglang/jit/sm86/` and
|
||
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
|
||
change lands.
|
||
|
||
## Payload
|
||
|
||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||
all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
|
||
re-checks). The serve script refuses to start with a missing shard.
|
||
|
||
## Client contract
|
||
|
||
- **Alias:** `frank-small` (the `id` at `/v1/models`).
|
||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||
as llama.cpp; models.yml sets `reasoningContentField:
|
||
"reasoning_content"`). `--reasoning-parser qwen3`.
|
||
- **Tools:** `--tool-call-parser qwen3_coder`.
|
||
- **Vision:** supported (in-checkpoint tower).
|
||
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
|
||
`--top-k` — sampling rides on the request; the harness sends the Qwen
|
||
recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
|
||
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
|
||
has no effort gate).
|
||
|
||
## Measured
|
||
|
||
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
|
||
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
|
||
above is what makes that compile succeed).
|
||
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
|
||
- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
|
||
- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of
|
||
live KV at once).
|
||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||
|
||
## Rollback
|
||
|
||
[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
|
||
remain intact on disk; its unit/env/serve trio is in `old/` after the
|
||
cutover.
|
||
|
||
# Citations
|
||
|
||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||
- SGLang: https://github.com/sgl-project/sglang
|
||
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
|