Frank
8551bb006b
frank-small: llama.cpp CUDA (master, sm_86) on MiMo Q6_K GGUF, 2 x 262144, asymmetric KV (f16 K + q8_0 V, flash-attn pinned for the V quant)
38 lines
1.7 KiB
Bash
38 lines
1.7 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-01 night (Frank): swapped from SGLang INT8-W8A16-AutoRound to the
|
|
# GGUF path specifically for the ASYMMETRIC KV contract — K unquantized
|
|
# (f16), V q8_0 — which only llama.cpp exposes per-tensor. Q6_K is 7.79 GiB
|
|
# of weights (above the fleet's 6-bit accuracy floor).
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 dim). K f16 = 16 KiB/token; V q8_0 ~ 8.5 KiB/token
|
|
# (34 B per 32-element block) => ~24.5 KiB/token. Two full native windows
|
|
# (2 x 262,144 = 524,288 tokens) cost ~12.6 GiB; weights 7.8 + KV 12.6 +
|
|
# compute buffers and CUDA graphs ~1.5-2 => ~22-23 GiB against the card's
|
|
# 24 GiB — this is the "two contexts with room for checkpoints" fit. If the
|
|
# graph capture OOMs, lower CTX_SIZE (262144 = two 131072 windows) rather
|
|
# than quantizing K.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf
|
|
|
|
# Fleet endpoint: same port and alias as the SGLang era; the harness
|
|
# models.yml id does not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two contexts at the model's native window each. llama.cpp divides
|
|
# --ctx-size across --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and
|
|
# without unified KV each slot gets its own cache.
|
|
CTX_SIZE=524288
|
|
PARALLEL=2
|
|
|
|
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
|
# q8_0 on V is only served by the flash-attention path; the serve script
|
|
# pins --flash-attn on and must never lose it.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|