Frank
35e24db7e6
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack. Motivation: harness compaction. frank-primary's contextWindow 262,144 compacts at 70% = 183,501 tokens, and the compaction request carries a 32,768-token summary budget -> 216,269 must fit ONE frank-small request. The 151,552 slot window could not hold it; the native window does. No spec-decode interaction: checkpoint ships zero mtp.* tensors (config lies), seam keeps every --speculative-* flag away from the engine.
7.7 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| reference | frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 | The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves. |
|
2026-10-01 |
frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
ACTIVE since 2026-10-01. Replaces frank-small-gemma4-12b (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port (7771), same alias (
frank-small), same client contract surface.
Identity
- Endpoint:
frank-smallat192.168.0.143:7771(Dankclaw). - Runtime: SGLang (latest) in a self-contained python3.12 venv at
/home/dank/GitPackages/sglang-cu128/(torch 2.13.0+cu130, transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root systemd unitscripts/frank-small.service(User=dank,LimitMEMLOCK=infinity) launched byscripts/serve-frank-small.shwith tunables fromscripts/frank-small.env. - Model:
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
Shape
- Architecture
qwen3_5hybrid: 32 layers = 8 full-attention (4 KV heads x 256 head_dim,partial_rotary_factor: 0.25) + 24 Gated DeltaNet.max_position_embeddings: 262144, and aqwen3_5_visiontower (27 layers) in the safetensors — no separate mmproj file. - Context: two 262,144-token slots at the model's native window
(
--context-length 262144,--max-running-requests 2) against a pinned 606,208-token pool (--max-total-tokens) — 2026-10-01 reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and a compaction request (frank-primary's trigger 183,501 + a 32,768-token summary budget = 216,269) now fits one request. A single request may grow to the whole pool — a 170,019-token prompt was verified at the 4-slot shape — the grid is the two-way concurrency contract. - KV:
fp8_e4m3— 16 KiB/token (8 attn layers x 2 x 4 heads x 256 dim x 1 B), pool 9.31 GiB. - The memory shape (the 4.5 GiB the naive math misses): the DeltaNet
ssm state pool is a first-class consumer. The engine autofills the
mamba slot count to spend a fixed ~5.1 GB budget regardless of dtype
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
pinned:
--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32= 795 MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and dies in capture, or later in the first long request once Triton's deferred kernel loads eat the remainder). Hence--disable-prefill-cuda-graph; decode graphs are untouched. - Quantization:
--quantization auto-round— the checkpoint'squantization_config.jsondeclaresquant_method: "auto-round"and SGLang loads it natively (vision tower,embed_tokensandlinear_attn.in_projstay fp16).
Spec method
None — and this is a checkpoint fact, not a tuning choice. The
text_config declares mtp_num_hidden_layers: 1, but the shipped
safetensors carry zero mtp.* tensors: verified 2026-10-01 against
model.safetensors.index.json for both this INT8 conversion and the base
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (32 layers, indices 0–31, no
mtp./nextn./layers.32 keys in either). With NEXTN enabled, SGLang
boots a 4.24 GiB draft model (Qwen3_5ForCausalLMMTP) from nothing and
decodes at accept len: 1.00, accept rate: 0.00 — every proposal rejected,
paying the 4-token verify tax for no token. The serve script passes no
--speculative-* flag, and the seam gates that.
CUDA 13 toolchain (the sharp edge)
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
flashinfer) compiles with a CUDA 13.0 nvcc extracted from debs into the
venv at cuda-nvcc/usr/local/cuda-13.0/ — the system nvcc is 12.4 (no
--compress-mode) and its headers clash with the 13.0 device passes.
Components (all from the ubuntu2404 CUDA repo, dpkg -x'd, not installed):
| Package | Supplies |
|---|---|
cuda-nvcc-13-0 (13.0.88) |
bin/nvcc, ptxas, cudafe++, … |
libnvvm-13-0 (13.0.48) |
nvvm/bin/cicc (the device compiler) |
cuda-cudart-dev-13-0 (13.0.88) |
include/cuda_runtime.h, lib64/stubs |
cuda-crt-13-0 (13.0.88) |
include/crt/host_runtime.h (2-arg __cudaLaunch) |
libnvptxcompiler-13-0 (13.0.88) |
static lib + nvPTXCompiler.h |
cuda-cccl-13-0 (13.0.50) |
cccl/cuda/... headers (libcudacxx) |
Three local modifications the tree carries:
bin/nvccis a shim overbin/nvcc.realadding-allow-unsupported-compiler(host GCC 16 > nvcc's max 13).targets/x86_64-linux/includeandlibare symlinks to../../../include/../../../lib64(the deb ships them as real near-empty dirs; the stub pass must see the 13.0 headers, not/usr/include12.4).include/cuda→cccl/cudasymlink (CCCL 3.x layout; flashinfer includes<cuda/functional>), andcrt/math_functions.hpatched withnoexceptonrsqrt/rsqrtf(glibc 2.42 declares themnoexcept(true); the unpatched 13.0 declarations are incompatible in the C++ host stub).
JIT caches: ~/.cache/sglang/jit/sm86/ and
~/.cache/sglang/.cache/flashinfer/0.6.18/86/ — clear both when a toolchain
change lands.
Payload
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
files under /home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/,
all LFS sha256-verified by scripts/download-frank-small.sh (--verify-only
re-checks). The serve script refuses to start with a missing shard.
Client contract
- Alias:
frank-small(theidat/v1/models). - Reasoning: extracted to
reasoning_content(live-verified 2026-10-01 — this SGLang build's qwen3 parser uses the same field name as llama.cpp; models.yml setsreasoningContentField: "reasoning_content").--reasoning-parser qwen3. - Tools:
--tool-call-parser qwen3_coder. - Vision: supported (in-checkpoint tower).
- Sampling: SGLang has NO server-wide
--temperature/--top-p/--top-k— sampling rides on the request; the harness sends the Qwen recipe (temp 0.7, top_p 0.8) via models.ymlcompat.extraBody. - Reasoning effort: the full OpenAI set is accepted (the chat template has no effort gate).
Measured
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain above is what makes that compile succeed).
- Decode 83.9 tok/s (no spec decode — nothing to accept).
- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of live KV at once).
- Vision: a 64x64 red square answers "Red" with
image_tokens: 64.
Rollback
frank-small-gemma4-12b — the llama.cpp Gemma4-12B QAT payload and binary
remain intact on disk; its unit/env/serve trio is in old/ after the
cutover.
Citations
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
- SGLang: https://github.com/sgl-project/sglang
- Serve script:
scripts/serve-frank-small.sh; tunables:scripts/frank-small.env