frank-large: roll back to gufo Qwen3.8-27B Q4_K_XL, one session x 262144 f16 KV
frank-small: llama.cpp CUDA (master, sm_86) on MiMo Q6_K GGUF, 2 x 262144, asymmetric KV (f16 K + q8_0 V, flash-attn pinned for the V quant)
This commit is contained in:
Frank committed
1 parent
e452a1afbf
commit
8551bb006b
11 files changed
+729
-816
No files matched your search
+29
-78
@@ -1,86 +1,37 @@
|
||||
# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
|
||||
# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
|
||||
# Dankclaw's RTX 3090, since 2026-10-01.
|
||||
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
||||
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K on the RTX 3090, :7771.
|
||||
#
|
||||
# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
|
||||
# python3.12 -m venv /home/dank/GitPackages/sglang-cu128
|
||||
# /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
|
||||
# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
|
||||
# self-contained and does not touch the host Python (3.14) or the ROCm
|
||||
# stack that serves frank-large.
|
||||
# 2026-10-01 night (Frank): swapped from SGLang INT8-W8A16-AutoRound to the
|
||||
# GGUF path specifically for the ASYMMETRIC KV contract — K unquantized
|
||||
# (f16), V q8_0 — which only llama.cpp exposes per-tensor. Q6_K is 7.79 GiB
|
||||
# of weights (above the fleet's 6-bit accuracy floor).
|
||||
#
|
||||
# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
|
||||
# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
|
||||
# safetensors shards, 12.0 GiB total. The vision tower
|
||||
# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
|
||||
# checkpoint, so the image path needs no separate projector file.
|
||||
# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
|
||||
# + 24 Gated DeltaNet, native 262144 window. text_config declares
|
||||
# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
|
||||
# tensors (verified against model.safetensors.index.json 2026-10-01 —
|
||||
# base BF16 and this INT8 conversion both omit the head), so there is no
|
||||
# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
|
||||
# model at 0% acceptance and pay the verify tax for nothing.
|
||||
#
|
||||
# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
|
||||
# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
|
||||
# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
|
||||
# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
|
||||
# take most of it after boot, and eager-prefill activations then OOM on
|
||||
# the first long request. The pool is therefore PINNED at 606,208 tokens
|
||||
# (2 x 262,144 native slots since the 2026-10-01 reslice; 4 x 151,552
|
||||
# before), which leaves ~0.8 GB for the prefill workspace; a 170k single
|
||||
# request and 4 concurrent 54k requests were verified at the 4-slot
|
||||
# shape.
|
||||
# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract); a client that sends a
|
||||
# temperature gets its own, and the harness applies per-model overrides
|
||||
# from models.yml.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
||||
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
||||
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 dim). K f16 = 16 KiB/token; V q8_0 ~ 8.5 KiB/token
|
||||
# (34 B per 32-element block) => ~24.5 KiB/token. Two full native windows
|
||||
# (2 x 262,144 = 524,288 tokens) cost ~12.6 GiB; weights 7.8 + KV 12.6 +
|
||||
# compute buffers and CUDA graphs ~1.5-2 => ~22-23 GiB against the card's
|
||||
# 24 GiB — this is the "two contexts with room for checkpoints" fit. If the
|
||||
# graph capture OOMs, lower CTX_SIZE (262144 = two 131072 windows) rather
|
||||
# than quantizing K.
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf
|
||||
|
||||
# SGLang runtime: python3.12 venv, self-contained CUDA stack.
|
||||
SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
|
||||
|
||||
# Model package (8 shards + configs, sha256-verified by
|
||||
# scripts/download-frank-small.sh).
|
||||
MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
|
||||
# Fleet identity and network. 7771 is this service's own endpoint on
|
||||
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
|
||||
# and what clients ask for -- the harness entry in models.yml names it.
|
||||
# Fleet endpoint: same port and alias as the SGLang era; the harness
|
||||
# models.yml id does not move.
|
||||
ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# Two slots at the model's native 262,144 window on fp8 KV. The pool is
|
||||
# pinned (see SIZING above): 606,208 tokens, and 2 x 262,144 = 524,288
|
||||
# leaves ~82k tokens of pool slack. Resliced 2026-10-01 from 4 x 151,552
|
||||
# so a compaction request (frank-primary's trigger 183,501 tokens + a
|
||||
# 32,768-token summary budget = 216,269) fits ONE request; the grid is
|
||||
# the two-way concurrency contract. A single request may still grow to
|
||||
# the whole pool (a 170k prompt was verified at the 4-slot shape).
|
||||
MEM_FRACTION=0.98
|
||||
MAX_TOTAL_TOKENS=606208
|
||||
CONTEXT_LENGTH=262144
|
||||
MAX_RUNNING=2
|
||||
KV_DTYPE=fp8_e4m3
|
||||
QUANTIZATION=auto-round
|
||||
# DeltaNet (mamba) recurrent states: the ssm state pool is the single
|
||||
# biggest hidden consumer on this card. The engine autofills the slot
|
||||
# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
|
||||
# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
|
||||
# bf16 dtype + 32 slots = 795 MiB.
|
||||
MAMBA_SSM_DTYPE=bfloat16
|
||||
# 32 slots is 16x the 2-way concurrency: enough for the radix mamba
|
||||
# cache to keep shared-prefix states, small enough to leave the KV pool
|
||||
# room.
|
||||
MAX_MAMBA_CACHE_SIZE=32
|
||||
# Two contexts at the model's native window each. llama.cpp divides
|
||||
# --ctx-size across --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and
|
||||
# without unified KV each slot gets its own cache.
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2
|
||||
|
||||
# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
|
||||
# grammar (both parsers per the SGLang Qwen3.5 cookbook).
|
||||
REASONING_PARSER=qwen3
|
||||
TOOLCALL_PARSER=qwen3_coder
|
||||
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
||||
# q8_0 on V is only served by the flash-attention path; the serve script
|
||||
# pins --flash-attn on and must never lose it.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
NGL=99
|
||||
Reference in new issue
Block a user