Files

T

Frank 934285c2b2 docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it

2026-10-02 03:46:50 -07:00

9.0 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)

RETIRED 2026-10-01, hours after cutover. Replaced by frank-small-llama-mimo-q6k — llama.cpp CUDA on the MiMo Q6_K GGUF, chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. The venv (/home/dank/GitPackages/sglang-cu128/) stays in place; the INT8 payload was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ on 2026-10-02 when the llama.cpp endpoint was two runtimes deep — recoverable until the trash is purged.

Identity

Endpoint: frank-small at 192.168.0.143:7771 (Dankclaw).
Runtime: SGLang (latest) in a self-contained python3.12 venv at /home/dank/GitPackages/sglang-cu128/ (torch 2.13.0+cu130, transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root systemd unit scripts/frank-small.service (User=dank, LimitMEMLOCK=infinity) launched by scripts/serve-frank-small.sh with tunables from scripts/frank-small.env.
Model: letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).

Shape

Architecture qwen3_5 hybrid: 32 layers = 8 full-attention (4 KV heads x 256 head_dim, partial_rotary_factor: 0.25) + 24 Gated DeltaNet. max_position_embeddings: 262144, and a qwen3_5_vision tower (27 layers) in the safetensors — no separate mmproj file.
Context: two 262,144-token slots at the model's native window (--context-length 262144, --max-running-requests 2) against a pinned 606,208-token pool (--max-total-tokens) — 2026-10-01 reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and a compaction request (frank-primary's trigger 183,501 + a 32,768-token summary budget = 216,269) now fits one request. A single request may grow to the whole pool — a 170,019-token prompt was verified at the 4-slot shape — the grid is the two-way concurrency contract. That compaction arithmetic is specific to a 262,144-token slot: the ACTIVE shape runs three slots of 180,224, which 183,501 tokens already exceeds — see What three slots cost in frank-small-llama-mimo-q5km.
KV: fp8_e4m3 — 16 KiB/token (8 attn layers x 2 x 4 heads x 256 dim x 1 B), pool 9.31 GiB.
The memory shape (the 4.5 GiB the naive math misses): the DeltaNet ssm state pool is a first-class consumer. The engine autofills the mamba slot count to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are pinned: --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32 = 795 MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and dies in capture, or later in the first long request once Triton's deferred kernel loads eat the remainder). Hence --disable-prefill-cuda-graph; decode graphs are untouched.
Quantization: --quantization auto-round — the checkpoint's quantization_config.json declares quant_method: "auto-round" and SGLang loads it natively (vision tower, embed_tokens and linear_attn.in_proj stay fp16).

Spec method

None — and this is a checkpoint fact, not a tuning choice. The text_config declares mtp_num_hidden_layers: 1, but the shipped safetensors carry zero mtp.* tensors: verified 2026-10-01 against model.safetensors.index.json for both this INT8 conversion and the base XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (32 layers, indices 0–31, no mtp./nextn./layers.32 keys in either). With NEXTN enabled, SGLang boots a 4.24 GiB draft model (Qwen3_5ForCausalLMMTP) from nothing and decodes at accept len: 1.00, accept rate: 0.00 — every proposal rejected, paying the 4-token verify tax for no token. The serve script passes no --speculative-* flag, and the seam gates that.

CUDA 13 toolchain (the sharp edge)

The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack, flashinfer) compiles with a CUDA 13.0 nvcc extracted from debs into the venv at cuda-nvcc/usr/local/cuda-13.0/ — the system nvcc is 12.4 (no --compress-mode) and its headers clash with the 13.0 device passes.

Components (all from the ubuntu2404 CUDA repo, dpkg -x'd, not installed):

Package	Supplies
`cuda-nvcc-13-0` (13.0.88)	`bin/nvcc`, `ptxas`, `cudafe++`, …
`libnvvm-13-0` (13.0.48)	`nvvm/bin/cicc` (the device compiler)
`cuda-cudart-dev-13-0` (13.0.88)	`include/cuda_runtime.h`, `lib64/stubs`
`cuda-crt-13-0` (13.0.88)	`include/crt/host_runtime.h` (2-arg `__cudaLaunch`)
`libnvptxcompiler-13-0` (13.0.88)	static lib + `nvPTXCompiler.h`
`cuda-cccl-13-0` (13.0.50)	`cccl/cuda/...` headers (libcudacxx)

Three local modifications the tree carries:

bin/nvcc is a shim over bin/nvcc.real adding -allow-unsupported-compiler (host GCC 16 > nvcc's max 13).
targets/x86_64-linux/include and lib are symlinks to ../../../include / ../../../lib64 (the deb ships them as real near-empty dirs; the stub pass must see the 13.0 headers, not /usr/include 12.4).
include/cuda → cccl/cuda symlink (CCCL 3.x layout; flashinfer includes <cuda/functional>), and crt/math_functions.h patched with noexcept on rsqrt/rsqrtf (glibc 2.42 declares them noexcept(true); the unpatched 13.0 declarations are incompatible in the C++ host stub).

JIT caches: ~/.cache/sglang/jit/sm86/ and ~/.cache/sglang/.cache/flashinfer/0.6.18/86/ — clear both when a toolchain change lands.

Payload

Eight safetensors shards (12,891,718,440 B total) + tokenizer and config files under /home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/, all LFS sha256-verified by old/download-frank-small-sglang-int8.sh (--verify-only re-checks) — the script moved to old/ when the endpoint cut over to llama.cpp + GGUF on 2026-10-01 night, and scripts/download-frank-small.sh now fetches the GGUF quant instead. The serve script refuses to start with a missing shard.

Client contract

Alias: frank-small (the id at /v1/models).
Reasoning: extracted to reasoning_content (live-verified 2026-10-01 — this SGLang build's qwen3 parser uses the same field name as llama.cpp; models.yml sets reasoningContentField: "reasoning_content"). --reasoning-parser qwen3.
Tools: --tool-call-parser qwen3_coder.
Vision: supported (in-checkpoint tower).
Sampling: SGLang has NO server-wide --temperature/--top-p/ --top-k — sampling rides on the request; the harness sends the Qwen recipe (temp 0.7, top_p 0.8) via models.yml compat.extraBody.
Reasoning effort: the full OpenAI set is accepted (the chat template has no effort gate).

Measured

Boot ~60 s with warm JIT caches (several minutes cold — the first boot compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain above is what makes that compile succeed).
Decode 83.9 tok/s (no spec decode — nothing to accept).
Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in 50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall incl. 8 decode tokens, ~2,056 t/s).
4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s (216k tokens of live KV at once). 2-slot shape: two concurrent 50,000-token requests both answered (28.8 s wall).
Compaction fit (the reslice's purpose): a 200,000-token prompt — well past frank-primary's 183,501-token compaction trigger (70% of its 262,144 window) — prefills and completes in ONE request.
Vision: a 64x64 red square answers "Red" with image_tokens: 64.

Rollback

frank-small-llama-mimo-q5km is the live shape two steps forward; the practical rollback today is frank-small-llama-mimo-q6k (payload at /home/dank/models/MiMo-Q6K/). This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/ on 2026-10-02; its unit/env/serve trio remains in old/.

Citations

Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
SGLang: https://github.com/sgl-project/sglang
Serve script: scripts/serve-frank-small.sh; tunables: scripts/frank-small.env

9.0 KiB Raw Blame History Unescape Escape