---
type: reference
title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
tags:
  - llm
  - frank-secondary
  - dankclaw
  - sglang
  - rtx3090
  - qwen
  - mtp
  - vision
  - int8
timestamp: 2026-10-01
---

# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)

> **RETIRED 2026-10-01, hours after cutover.** Replaced by
> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
> payload was staged to
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
> is purged.

## Identity

- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
  systemd unit `scripts/frank-secondary.service` (`User=dank`,
  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
  tunables from `scripts/frank-secondary.env`.
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).

## Shape

- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
  tower (27 layers) in the safetensors — no separate mmproj file.
- **Context:** two 262,144-token slots at the model's native window
  (`--context-length 262144`, `--max-running-requests 2`) against a
  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
  ~82k tokens of pool slack, and a compaction request (frank-primary's
  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
  one request. A single request may grow to the whole pool — a
  170,019-token prompt was verified at the 4-slot shape — the grid is
  the two-way concurrency contract. **That compaction arithmetic is
  specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
  180,224, which 183,501 tokens already exceeds — see *What three slots
  cost* in [[frank-secondary-llama-mimo-q5km]].
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
  dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
  ssm state pool is a first-class consumer. The engine autofills the
  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
  dies in capture, or later in the first long request once Triton's
  deferred kernel loads eat the remainder). Hence
  `--disable-prefill-cuda-graph`; decode graphs are untouched.
- **Quantization:** `--quantization auto-round` — the checkpoint's
  `quantization_config.json` declares `quant_method: "auto-round"` and
  SGLang loads it natively (vision tower, `embed_tokens` and
  `linear_attn.in_proj` stay fp16).

## Spec method

**None — and this is a checkpoint fact, not a tuning choice.** The
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
`model.safetensors.index.json` for both this INT8 conversion and the base
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
paying the 4-token verify tax for no token. The serve script passes no
`--speculative-*` flag, and the seam gates that.

## CUDA 13 toolchain (the sharp edge)

The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
`--compress-mode`) and its headers clash with the 13.0 device passes.

Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):

| Package | Supplies |
|---|---|
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |

Three local modifications the tree carries:

1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
   `../../../include` / `../../../lib64` (the deb ships them as real
   near-empty dirs; the stub pass must see the 13.0 headers, not
   `/usr/include` 12.4).
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
   the unpatched 13.0 declarations are incompatible in the C++ host stub).

JIT caches: `~/.cache/sglang/jit/sm86/` and
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
change lands.

## Payload

Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
cut over to llama.cpp + GGUF on 2026-10-01 night, and
`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
serve script refuses to start with a missing shard.

## Client contract

- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
  as llama.cpp; models.yml sets `reasoningContentField:
  "reasoning_content"`). `--reasoning-parser qwen3`.
- **Tools:** `--tool-call-parser qwen3_coder`.
- **Vision:** supported (in-checkpoint tower).
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
  `--top-k` — sampling rides on the request; the harness sends the Qwen
  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
  has no effort gate).

## Measured

- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
  above is what makes that compile succeed).
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
  50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
  incl. 8 decode tokens, ~2,056 t/s).
- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
  (216k tokens of live KV at once). 2-slot shape: two concurrent
  50,000-token requests both answered (28.8 s wall).
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
  well past frank-primary's 183,501-token compaction trigger (70% of its
  262,144 window) — prefills and completes in ONE request.
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.

## Rollback

[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
Gemma4-12B QAT payload, was staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
2026-10-02; its unit/env/serve trio remains in `old/`.

# Citations

- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
- SGLang: https://github.com/sgl-project/sglang
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`