# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
# Dankclaw's RTX 3090, since 2026-10-01.
#
# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
#   python3.12 -m venv /home/dank/GitPackages/sglang-cu128
#   /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
# self-contained and does not touch the host Python (3.14) or the ROCm
# stack that serves frank-large.
#
# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
# safetensors shards, 12.0 GiB total. The vision tower
# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
# checkpoint, so the image path needs no separate projector file.
# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
# + 24 Gated DeltaNet, native 262144 window. text_config declares
# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
# tensors (verified against model.safetensors.index.json 2026-10-01 —
# base BF16 and this INT8 conversion both omit the head), so there is no
# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
# model at 0% acceptance and pay the verify tax for nothing.
#
# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
# take most of it after boot, and eager-prefill activations then OOM on
# the first long request. The pool is therefore PINNED at 606,208 tokens
# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
# single request and 4 concurrent 54k requests are verified against it.
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
#
# Sampling: greedy by default (the fleet contract); a client that sends a
# temperature gets its own, and the harness applies per-model overrides
# from models.yml.
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.

# SGLang runtime: python3.12 venv, self-contained CUDA stack.
SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang

# Model package (8 shards + configs, sha256-verified by
# scripts/download-frank-small.sh).
MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound

# Fleet identity and network. 7771 is this service's own endpoint on
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
# and what clients ask for -- the harness entry in models.yml names it.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
# tokens = 4 x 151,552, so the slot window is 151,552. A single request
# may still grow to the whole pool (a 170k prompt is verified); the grid
# is the four-way concurrency contract.
MEM_FRACTION=0.98
MAX_TOTAL_TOKENS=606208
CONTEXT_LENGTH=151552
MAX_RUNNING=4
KV_DTYPE=fp8_e4m3
QUANTIZATION=auto-round
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
# biggest hidden consumer on this card. The engine autofills the slot
# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
# bf16 dtype + 32 slots = 795 MiB.
MAMBA_SSM_DTYPE=bfloat16
# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
# to keep shared-prefix states, small enough to leave the KV pool room.
MAX_MAMBA_CACHE_SIZE=32

# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
# grammar (both parsers per the SGLang Qwen3.5 cookbook).
REASONING_PARSER=qwen3
TOOLCALL_PARSER=qwen3_coder