# frank-small — llama.cpp (master, CUDA sm_86) serving
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
#
# 2026-10-01 night (Frank): one quant step down from the Q6_K file this
# endpoint ran for a few hours, on the same runtime and the same shape — the
# llama.cpp GGUF path is kept for the ASYMMETRIC KV contract (K unquantized
# f16, V q8_0) that no other runtime on this host exposes per-tensor, and the
# freed 0.86 GiB of weights is what keeps the two full windows bootable with
# margin for CUDA graphs.
#
# KV arithmetic: attention KV exists only in the 8 full-attention layers
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token; V q8_0 = 8.5 KiB/token
# (34 B per 32-element block) => 24.5 KiB/token. Two full native windows
# (2 x 262,144 = 524,288 tokens) cost 12.5 GiB; weights 6.40 + KV 12.5 + the
# fixed Gated DeltaNet state (~25 MiB/slot) + compute buffers and CUDA graphs
# ~1.5-2 => ~20.5 GiB against the card's 24 GiB. Q6_K measured 20.9/24.6 here,
# which is the same arithmetic with 7.26 GiB of weights.
#
# Why V is not f16: full-precision V costs 16 KiB/token instead of 8.5, i.e.
# +3.96 GiB on a 524,288-token pool — 24.5 GiB total, past the card. If the
# graph capture OOMs, lower CTX_SIZE (262144 = two 131,072 windows) rather
# than quantizing K.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf

# Fleet endpoint: same port and alias as every frank-small before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# Two contexts at the model's native window each. llama.cpp divides
# --ctx-size across --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and
# without unified KV each slot gets its own cache.
CTX_SIZE=524288
PARALLEL=2

# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
# q8_0 on V is only served by the flash-attention path; the serve script
# pins --flash-attn on and must never lose it.
CACHE_TYPE_K=f16
CACHE_TYPE_V=q8_0
NGL=99