# frank-small — llama.cpp (master, CUDA sm_86) serving
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
#
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
# window shrinks — see the budget below.
#
# Two measurements on this card, this runtime, price the terms:
#   2 x 262,144, no projector: 20,190 MiB used / 4,386 free
#     KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
#     DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
#   3 x 196,608, projector on: 22,972 MiB used / 1,153 free
#     KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
#
# KV arithmetic: attention KV exists only in the 8 full-attention layers
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
# is one 293 MiB step — that is the ladder to walk if this moves again. The
# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
#
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
# the tensor the asymmetric-KV contract exists to protect.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf

# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf

# Fleet endpoint: same port and alias as every frank-small before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# Three independent conversations. llama.cpp divides --ctx-size across
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
# cannot use unified KV, so each slot gets its own cache and its own window.
# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
# third slot: a longer single prompt needs compaction instead of one shot.
# Consequence worth knowing: frank-primary's compaction trigger is 183,501
# tokens, which no longer fits a frank-small slot — see the runbook's
# "three slots vs compaction" note.
CTX_SIZE=540672
PARALLEL=3

# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
# q8_0 on V is only served by the flash-attention path; the serve script
# pins --flash-attn on and must never lose it.
CACHE_TYPE_K=f16
CACHE_TYPE_V=q8_0
NGL=99