# frank-small — llama.cpp (master, CUDA sm_86) serving
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K on the RTX 3090, :7771.
#
# 2026-10-01 night (Frank): swapped from SGLang INT8-W8A16-AutoRound to the
# GGUF path specifically for the ASYMMETRIC KV contract — K unquantized
# (f16), V q8_0 — which only llama.cpp exposes per-tensor. Q6_K is 7.79 GiB
# of weights (above the fleet's 6-bit accuracy floor).
#
# KV arithmetic: attention KV exists only in the 8 full-attention layers
# (4 KV heads x 256 dim). K f16 = 16 KiB/token; V q8_0 ~ 8.5 KiB/token
# (34 B per 32-element block) => ~24.5 KiB/token. Two full native windows
# (2 x 262,144 = 524,288 tokens) cost ~12.6 GiB; weights 7.8 + KV 12.6 +
# compute buffers and CUDA graphs ~1.5-2 => ~22-23 GiB against the card's
# 24 GiB — this is the "two contexts with room for checkpoints" fit. If the
# graph capture OOMs, lower CTX_SIZE (262144 = two 131072 windows) rather
# than quantizing K.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf

# Fleet endpoint: same port and alias as the SGLang era; the harness
# models.yml id does not move.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# Two contexts at the model's native window each. llama.cpp divides
# --ctx-size across --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and
# without unified KV each slot gets its own cache.
CTX_SIZE=524288
PARALLEL=2

# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
# q8_0 on V is only served by the flash-attention path; the serve script
# pins --flash-attn on and must never lose it.
CACHE_TYPE_K=f16
CACHE_TYPE_V=q8_0
NGL=99