frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
Frank committed
1 parent
7da504a651
commit
0c5e054fb8
12 files changed
+1005
-328
No files matched your search
+60
-109
@@ -1,37 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
|
||||
# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
|
||||
# Serve letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound as
|
||||
# frank-small on Dankclaw's RTX 3090 via SGLang.
|
||||
#
|
||||
# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
|
||||
# weights plus the BF16 vision projector and q8_0 KV inside the card's
|
||||
# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
|
||||
# belongs to the halogen engine that serves frank-large.
|
||||
# Topology: the whole checkpoint (12.0 GiB, INT8 W8A16 AutoRound) lives in
|
||||
# VRAM; the 3090's 24 GiB holds weights + a ~11.8 GiB fp8 KV pool at
|
||||
# --mem-fraction-static 0.98. The host CPU only tokenizes.
|
||||
#
|
||||
# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
|
||||
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
|
||||
# passed via --model-draft with --spec-type draft-mtp: the draft context
|
||||
# loads the assistant model. The same --spec-type value chains the
|
||||
# draftless ngram-mod matcher, which attaches unconditionally; two
|
||||
# context-backed drafters cannot share one process, so this chain is the
|
||||
# only shape that carries both.
|
||||
# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
|
||||
# DeltaNet, native 262144 window, and a vision tower (qwen3_5_vision)
|
||||
# baked into the safetensors: no separate projector file, the image path
|
||||
# is in the checkpoint. The config declares an MTP layer, but the shipped
|
||||
# weights carry none (no mtp.* tensors in the index), so this service runs
|
||||
# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
|
||||
# 4.24 GiB ghost model drafting at 0% acceptance.
|
||||
#
|
||||
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
|
||||
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
|
||||
# maximises the TOTAL pool instead, which would divide a smaller window
|
||||
# across slots. The 4-slot layout is measured to fit the card (see the
|
||||
# SIZING note in frank-small.env); more slots would fail the KV allocation.
|
||||
# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool,
|
||||
# which the engine autofits around the weights AND the DeltaNet ssm state
|
||||
# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
|
||||
# pinned (bf16, 32 slots). With the grid at its measured fit the card has
|
||||
# no headroom for prefill CUDA-graph capture, so prefill runs eager
|
||||
# (--disable-prefill-cuda-graph); decode graphs are untouched.
|
||||
#
|
||||
# Vision + audio: --mmproj loads the BF16 projector, which also carries
|
||||
# an audio encoder — input_audio content parts transcribe verbatim
|
||||
# (verified 2026-09-17). The fleet's image path lives here, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
# models.yml. --jinja is pinned even though this build defaults it on: the
|
||||
# model's own template supplies reasoning_effort, preserve_thinking and the
|
||||
# tool-call grammar, and a future build that flips that default must not
|
||||
# silently take tools away.
|
||||
# Sampling: greedy by default (the fleet contract); clients that send their
|
||||
# own temperature get their own values.
|
||||
#
|
||||
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
|
||||
set -euo pipefail
|
||||
@@ -46,101 +37,61 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
|
||||
: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
|
||||
: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
|
||||
# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
|
||||
# payload, which frank-small served on 2026-09-16/17 and which no longer
|
||||
# exists on disk. frank-small.env has always overridden it, so the drift was
|
||||
# invisible — but a missing or unreadable env file would have left this
|
||||
# script pointing at a payload that is not there, turning a config problem
|
||||
# into a "missing asset" that names the wrong model.
|
||||
: "${DRAFT_FILE:=}"
|
||||
: "${SGLANG_BIN:=/home/dank/GitPackages/sglang-cu128/bin/sglang}"
|
||||
: "${MODEL_DIR:=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
|
||||
: "${ALIAS:=frank-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
: "${CTX_SIZE:=}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
|
||||
# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
|
||||
# busy-wait level (build default 50).
|
||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||
: "${POLL_LEVEL:=50}"
|
||||
: "${NUM_THREADS:=8}"
|
||||
: "${THREAD_BATCH:=8}"
|
||||
: "${NGL:=999}"
|
||||
: "${FLASH_ATTN:=on}"
|
||||
: "${SPEC_TYPE:=draft-mtp}"
|
||||
: "${TEMPERATURE:=0.0}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${REASONING_BUDGET:=-1}"
|
||||
: "${MEM_FRACTION:=0.98}"
|
||||
: "${CONTEXT_LENGTH:=151552}"
|
||||
: "${MAX_TOTAL_TOKENS:=606208}"
|
||||
: "${MAX_RUNNING:=4}"
|
||||
: "${KV_DTYPE:=fp8_e4m3}"
|
||||
: "${QUANTIZATION:=auto-round}"
|
||||
: "${REASONING_PARSER:=qwen3}"
|
||||
: "${TOOLCALL_PARSER:=qwen3_coder}"
|
||||
: "${MAMBA_SSM_DTYPE:=bfloat16}"
|
||||
: "${MAX_MAMBA_CACHE_SIZE:=32}"
|
||||
|
||||
# Fail loudly rather than half-loading a model into VRAM: a missing file here
|
||||
# is a download that has not finished, and the unit's restart policy would
|
||||
# otherwise turn that into a crash loop that reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
|
||||
for path in "$SGLANG_BIN" "$MODEL_DIR/model.safetensors.index.json" \
|
||||
"$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin/nvcc"; do
|
||||
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
||||
done
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
[[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
|
||||
shard_count=$(find "$MODEL_DIR" -maxdepth 1 -name 'model-*.safetensors' | wc -l)
|
||||
if [[ "$shard_count" -ne 8 ]]; then
|
||||
echo "frank-small: expected 8 safetensors shards in $MODEL_DIR, found $shard_count" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
|
||||
fi
|
||||
# FlashInfer JITs its attention kernels with nvcc at boot. The host's
|
||||
# /usr/bin/nvcc is CUDA 12.4 and dies on `--compress-mode` (a 12.8+
|
||||
# option), so the venv carries a matching CUDA 13.0 nvcc (extracted from
|
||||
# the cuda-nvcc-13-0 deb); it must shadow the system one on PATH.
|
||||
export PATH="$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin:$PATH"
|
||||
|
||||
SPEC_ARGS=()
|
||||
if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
|
||||
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
fi
|
||||
|
||||
DRAFT_ARGS=()
|
||||
if [[ -n "${DRAFT_FILE:-}" ]]; then
|
||||
DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
|
||||
fi
|
||||
|
||||
# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
|
||||
# whatever the weights, projector, draft context and compute buffers leave
|
||||
# on the card; the engine prints the chosen value at startup.
|
||||
if [[ -n "${CTX_SIZE:-}" ]]; then
|
||||
CTX_ARGS=(--ctx-size "$CTX_SIZE")
|
||||
else
|
||||
CTX_ARGS=(--fit on)
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL_DIR/$MODEL_FILE" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
"${DRAFT_ARGS[@]}" \
|
||||
--alias "$ALIAS" \
|
||||
# Sampling: SGLang has no server-wide sampling defaults; the harness sends
|
||||
# its per-model recipe from models.yml (compat.extraBody), and other clients
|
||||
# get the engine's own defaults.
|
||||
exec "$SGLANG_BIN" serve \
|
||||
--model-path "$MODEL_DIR" \
|
||||
--served-model-name "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
--gpu-layers "$NGL" \
|
||||
--flash-attn "$FLASH_ATTN" \
|
||||
"${CTX_ARGS[@]}" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$NUM_THREADS" \
|
||||
--threads-batch "$THREAD_BATCH" \
|
||||
--poll "$POLL_LEVEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
--temp "$TEMPERATURE" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget "$REASONING_BUDGET"
|
||||
--mem-fraction-static "$MEM_FRACTION" \
|
||||
--context-length "$CONTEXT_LENGTH" \
|
||||
--max-running-requests "$MAX_RUNNING" \
|
||||
--kv-cache-dtype "$KV_DTYPE" \
|
||||
--quantization "$QUANTIZATION" \
|
||||
--reasoning-parser "$REASONING_PARSER" \
|
||||
--tool-call-parser "$TOOLCALL_PARSER" \
|
||||
--mamba-ssm-dtype "$MAMBA_SSM_DTYPE" \
|
||||
--max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE" \
|
||||
--disable-prefill-cuda-graph \
|
||||
--max-total-tokens "$MAX_TOTAL_TOKENS"
|
||||
Reference in new issue
Block a user