Frank
35e24db7e6
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack. Motivation: harness compaction. frank-primary's contextWindow 262,144 compacts at 70% = 183,501 tokens, and the compaction request carries a 32,768-token summary budget -> 216,269 must fit ONE frank-small request. The 151,552 slot window could not hold it; the native window does. No spec-decode interaction: checkpoint ships zero mtp.* tensors (config lies), seam keeps every --speculative-* flag away from the engine.
87 lines
4.2 KiB
Bash
87 lines
4.2 KiB
Bash
# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
|
|
# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
|
|
# Dankclaw's RTX 3090, since 2026-10-01.
|
|
#
|
|
# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
|
|
# python3.12 -m venv /home/dank/GitPackages/sglang-cu128
|
|
# /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
|
|
# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
|
|
# self-contained and does not touch the host Python (3.14) or the ROCm
|
|
# stack that serves frank-large.
|
|
#
|
|
# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
|
|
# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
|
|
# safetensors shards, 12.0 GiB total. The vision tower
|
|
# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
|
|
# checkpoint, so the image path needs no separate projector file.
|
|
# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
|
|
# + 24 Gated DeltaNet, native 262144 window. text_config declares
|
|
# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
|
|
# tensors (verified against model.safetensors.index.json 2026-10-01 —
|
|
# base BF16 and this INT8 conversion both omit the head), so there is no
|
|
# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
|
|
# model at 0% acceptance and pay the verify tax for nothing.
|
|
#
|
|
# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
|
|
# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
|
|
# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
|
|
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
|
|
# take most of it after boot, and eager-prefill activations then OOM on
|
|
# the first long request. The pool is therefore PINNED at 606,208 tokens
|
|
# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
|
|
# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
|
|
# request and 4 concurrent 54k requests were verified at the 4-slot
|
|
# shape.
|
|
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# SGLang runtime: python3.12 venv, self-contained CUDA stack.
|
|
SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
|
|
|
|
# Model package (8 shards + configs, sha256-verified by
|
|
# scripts/download-frank-small.sh).
|
|
MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
|
|
# and what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two slots at the model's native 262,144 window on fp8 KV. The pool is
|
|
# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
|
|
# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
|
|
# so a compaction request (frank-primary's trigger 183,501 tokens + a
|
|
# 32,768-token summary budget = 216,269) fits ONE request; the grid is
|
|
# the two-way concurrency contract. A single request may still grow to
|
|
# the whole pool (a 170k prompt was verified at the 4-slot shape).
|
|
MEM_FRACTION=0.98
|
|
MAX_TOTAL_TOKENS=606208
|
|
CONTEXT_LENGTH=262144
|
|
MAX_RUNNING=2
|
|
KV_DTYPE=fp8_e4m3
|
|
QUANTIZATION=auto-round
|
|
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
|
|
# biggest hidden consumer on this card. The engine autofills the slot
|
|
# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
|
|
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
|
|
# bf16 dtype + 32 slots = 795 MiB.
|
|
MAMBA_SSM_DTYPE=bfloat16
|
|
# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
|
|
# cache to keep shared-prefix states, small enough to leave the KV pool
|
|
# room.
|
|
MAX_MAMBA_CACHE_SIZE=32
|
|
|
|
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
|
|
# grammar (both parsers per the SGLang Qwen3.5 cookbook).
|
|
REASONING_PARSER=qwen3
|
|
TOOLCALL_PARSER=qwen3_coder
|