# frank-small — llama.cpp (master, CUDA sm_86) serving
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
#
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
# window shrinks — see the budget below.
#
# Measured on this card, this runtime, 2 x 262,144 (2026-10-02, steady state):
# 20,190 MiB used of 24,576. KV at that pool was 524,288 tokens x 24.5 KiB =
# 12,800 MiB, so everything that is NOT KV — weights 6,557 + Gated DeltaNet
# state ~25/slot + compute buffers and CUDA graphs ~800 — is 7,390 MiB.
#
# KV arithmetic (unchanged): attention KV exists only in the 8 full-attention
# layers (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 =
# 8.5 KiB/token (34 B per 32-element block) => 24.5 KiB/token. Three slots at
# 196,608 = 589,824 tokens cost 14,113 MiB. Add the non-KV 7,390, a third
# DeltaNet state (~25) and the vision tower + its graph (~1,030: 876 MiB of
# mmproj weights plus buffer) => ~22.5 GiB used, ~2.0 GiB free — the bottom of
# the requested band. Step CTX_SIZE down by 12,288 (=4096/slot) if measurement
# says tighter, up if it says looser; the native 262,144 per slot is NOT
# reachable three times (3 x 262,144 x 24.5 KiB = 18.4 GiB of KV alone).
#
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
# the tensor the asymmetric-KV contract exists to protect.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf

# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf

# Fleet endpoint: same port and alias as every frank-small before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# Three independent conversations. llama.cpp divides --ctx-size across
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
# cannot use unified KV, so each slot gets its own cache and its own window.
# 196,608 per slot is BELOW the native 262,144: a single request longer than
# ~196k tokens now needs compaction instead of one shot (the 2026-10-01
# 2 x 262,144 shape was chosen to hold a 216,269-token compaction request;
# three contexts is the trade Frank asked for).
CTX_SIZE=589824
PARALLEL=3

# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
# q8_0 on V is only served by the flash-attention path; the serve script
# pins --flash-attn on and must never lose it.
CACHE_TYPE_K=f16
CACHE_TYPE_V=q8_0
NGL=99