frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
Frank committed
1 parent
7da504a651
commit
0c5e054fb8
12 files changed
+1005
-328
No files matched your search
+60
-98
@@ -1,46 +1,36 @@
|
||||
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
||||
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
||||
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
||||
# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
|
||||
# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
|
||||
# Dankclaw's RTX 3090, since 2026-10-01.
|
||||
#
|
||||
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
||||
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
||||
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
# cmake --build build-cuda --target llama-server -j
|
||||
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
||||
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
||||
# CUDA (see the CUDA lesson in AGENTS.md).
|
||||
# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
|
||||
# python3.12 -m venv /home/dank/GitPackages/sglang-cu128
|
||||
# /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
|
||||
# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
|
||||
# self-contained and does not touch the host Python (3.14) or the ROCm
|
||||
# stack that serves frank-large.
|
||||
#
|
||||
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
||||
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
||||
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
||||
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
||||
# only a 1024-position window, which is why the KV pool is a fraction of
|
||||
# the full-ctx x all-layers number.
|
||||
# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
|
||||
# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
|
||||
# safetensors shards, 12.0 GiB total. The vision tower
|
||||
# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
|
||||
# checkpoint, so the image path needs no separate projector file.
|
||||
# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
|
||||
# + 24 Gated DeltaNet, native 262144 window. text_config declares
|
||||
# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
|
||||
# tensors (verified against model.safetensors.index.json 2026-10-01 —
|
||||
# base BF16 and this INT8 conversion both omit the head), so there is no
|
||||
# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
|
||||
# model at 0% acceptance and pay the verify tax for nothing.
|
||||
#
|
||||
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
||||
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
||||
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
||||
# context loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
||||
# is the only legal shape that carries both.
|
||||
#
|
||||
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
||||
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
||||
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
||||
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
||||
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
||||
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
||||
# which would shrink each slot below the full 262144 window once
|
||||
# parallel > 1.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
||||
# params; it carries an audio encoder too). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
||||
# WORKS: input_audio content parts transcribe verbatim (verified
|
||||
# 2026-09-17 with a 16k mono WAV).
|
||||
# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
|
||||
# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
|
||||
# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
|
||||
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
|
||||
# take most of it after boot, and eager-prefill activations then OOM on
|
||||
# the first long request. The pool is therefore PINNED at 606,208 tokens
|
||||
# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
|
||||
# single request and 4 concurrent 54k requests are verified against it.
|
||||
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
@@ -50,69 +40,41 @@
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
|
||||
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
||||
# FA_ALL_QUANTS=ON.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
||||
# SGLang runtime: python3.12 venv, self-contained CUDA stack.
|
||||
SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
|
||||
|
||||
# Model package. The MTP sidecar file is required for speculative decoding.
|
||||
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
||||
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
||||
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
||||
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
||||
# Model package (8 shards + configs, sha256-verified by
|
||||
# scripts/download-frank-small.sh).
|
||||
MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
||||
# what clients ask for -- the harness entry in models.yml names it.
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
|
||||
# and what clients ask for -- the harness entry in models.yml names it.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
||||
# pool; --parallel divides it). See the SIZING note above for the fit
|
||||
# arithmetic; a fifth slot does not fit.
|
||||
CTX_SIZE=1048576
|
||||
PARALLEL=4
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
|
||||
# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
|
||||
# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
|
||||
# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
|
||||
# model, so the >12.8 GiB CACHE_RAM floor does not apply.
|
||||
CACHE_RAM=2048
|
||||
# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
|
||||
# tokens = 4 x 151,552, so the slot window is 151,552. A single request
|
||||
# may still grow to the whole pool (a 170k prompt is verified); the grid
|
||||
# is the four-way concurrency contract.
|
||||
MEM_FRACTION=0.98
|
||||
MAX_TOTAL_TOKENS=606208
|
||||
CONTEXT_LENGTH=151552
|
||||
MAX_RUNNING=4
|
||||
KV_DTYPE=fp8_e4m3
|
||||
QUANTIZATION=auto-round
|
||||
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
|
||||
# biggest hidden consumer on this card. The engine autofills the slot
|
||||
# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
|
||||
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
|
||||
# bf16 dtype + 32 slots = 795 MiB.
|
||||
MAMBA_SSM_DTYPE=bfloat16
|
||||
# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
|
||||
# to keep shared-prefix states, small enough to leave the KV pool room.
|
||||
MAX_MAMBA_CACHE_SIZE=32
|
||||
|
||||
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
||||
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
||||
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
||||
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
||||
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
||||
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
||||
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
||||
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# Every layer on the 3090, with flash attention (required for q8 KV).
|
||||
NGL=999
|
||||
FLASH_ATTN=on
|
||||
|
||||
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
||||
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
||||
# explicit here (both were the implicit build defaults: 3 and 50).
|
||||
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
||||
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
||||
# costs more attention re-read at long context than the extra accepted
|
||||
# token earns. Keep 3/50.
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
POLL_LEVEL=50
|
||||
|
||||
# Greedy default (fleet contract); reasoning is extracted into
|
||||
# reasoning_content with an unlimited budget (the model decides).
|
||||
TEMPERATURE=0.0
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
|
||||
# grammar (both parsers per the SGLang Qwen3.5 cookbook).
|
||||
REASONING_PARSER=qwen3
|
||||
TOOLCALL_PARSER=qwen3_coder
|
||||
Reference in new issue
Block a user