Frank
0c5e054fb8
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
81 lines
3.9 KiB
Bash
81 lines
3.9 KiB
Bash
# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
|
|
# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
|
|
# Dankclaw's RTX 3090, since 2026-10-01.
|
|
#
|
|
# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
|
|
# python3.12 -m venv /home/dank/GitPackages/sglang-cu128
|
|
# /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
|
|
# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
|
|
# self-contained and does not touch the host Python (3.14) or the ROCm
|
|
# stack that serves frank-large.
|
|
#
|
|
# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
|
|
# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
|
|
# safetensors shards, 12.0 GiB total. The vision tower
|
|
# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
|
|
# checkpoint, so the image path needs no separate projector file.
|
|
# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
|
|
# + 24 Gated DeltaNet, native 262144 window. text_config declares
|
|
# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
|
|
# tensors (verified against model.safetensors.index.json 2026-10-01 —
|
|
# base BF16 and this INT8 conversion both omit the head), so there is no
|
|
# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
|
|
# model at 0% acceptance and pay the verify tax for nothing.
|
|
#
|
|
# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
|
|
# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
|
|
# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
|
|
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
|
|
# take most of it after boot, and eager-prefill activations then OOM on
|
|
# the first long request. The pool is therefore PINNED at 606,208 tokens
|
|
# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
|
|
# single request and 4 concurrent 54k requests are verified against it.
|
|
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# SGLang runtime: python3.12 venv, self-contained CUDA stack.
|
|
SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
|
|
|
|
# Model package (8 shards + configs, sha256-verified by
|
|
# scripts/download-frank-small.sh).
|
|
MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
|
|
# and what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
|
|
# tokens = 4 x 151,552, so the slot window is 151,552. A single request
|
|
# may still grow to the whole pool (a 170k prompt is verified); the grid
|
|
# is the four-way concurrency contract.
|
|
MEM_FRACTION=0.98
|
|
MAX_TOTAL_TOKENS=606208
|
|
CONTEXT_LENGTH=151552
|
|
MAX_RUNNING=4
|
|
KV_DTYPE=fp8_e4m3
|
|
QUANTIZATION=auto-round
|
|
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
|
|
# biggest hidden consumer on this card. The engine autofills the slot
|
|
# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
|
|
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
|
|
# bf16 dtype + 32 slots = 795 MiB.
|
|
MAMBA_SSM_DTYPE=bfloat16
|
|
# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
|
|
# to keep shared-prefix states, small enough to leave the KV pool room.
|
|
MAX_MAMBA_CACHE_SIZE=32
|
|
|
|
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
|
|
# grammar (both parsers per the SGLang Qwen3.5 cookbook).
|
|
REASONING_PARSER=qwen3
|
|
TOOLCALL_PARSER=qwen3_coder
|