Frank
a02850203d
CTX_SIZE 589824 / PARALLEL 3 -> three independent 196,608-token conversations (a mamba hybrid cannot share one KV cache, so per-slot window = pool/slots; 3 x 262,144 would need 18.4 GiB of KV alone and busts the card). Vision comes back via bartowski's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf (918,166,048 B, sha256 ff348f31..., fetched + verified by scripts/download-frank-small.sh): the Q5_K_M GGUF ships no v.* tensors, so --mmproj IS the image path. MMPROJ_FILE without a colon default so an explicit empty value is a real text-only opt-out, and the deployer now gates on the projector exactly as the serve script does. Budget is measured, not guessed: the 2 x 262,144 shape used 20,190/24,576 MiB, which prices non-KV cost at 7,390 MiB. Three slots + projector land near 22.5 GiB, i.e. ~2 GiB spare - the bottom of the requested band, to be confirmed by measurement and laddered if it lands outside 2-3 GiB.
56 lines
2.9 KiB
Bash
56 lines
2.9 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
|
|
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
|
|
# window shrinks — see the budget below.
|
|
#
|
|
# Measured on this card, this runtime, 2 x 262,144 (2026-10-02, steady state):
|
|
# 20,190 MiB used of 24,576. KV at that pool was 524,288 tokens x 24.5 KiB =
|
|
# 12,800 MiB, so everything that is NOT KV — weights 6,557 + Gated DeltaNet
|
|
# state ~25/slot + compute buffers and CUDA graphs ~800 — is 7,390 MiB.
|
|
#
|
|
# KV arithmetic (unchanged): attention KV exists only in the 8 full-attention
|
|
# layers (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 =
|
|
# 8.5 KiB/token (34 B per 32-element block) => 24.5 KiB/token. Three slots at
|
|
# 196,608 = 589,824 tokens cost 14,113 MiB. Add the non-KV 7,390, a third
|
|
# DeltaNet state (~25) and the vision tower + its graph (~1,030: 876 MiB of
|
|
# mmproj weights plus buffer) => ~22.5 GiB used, ~2.0 GiB free — the bottom of
|
|
# the requested band. Step CTX_SIZE down by 12,288 (=4096/slot) if measurement
|
|
# says tighter, up if it says looser; the native 262,144 per slot is NOT
|
|
# reachable three times (3 x 262,144 x 24.5 KiB = 18.4 GiB of KV alone).
|
|
#
|
|
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
|
|
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
|
|
# the tensor the asymmetric-KV contract exists to protect.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
|
|
|
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
|
|
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Three independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 196,608 per slot is BELOW the native 262,144: a single request longer than
|
|
# ~196k tokens now needs compaction instead of one shot (the 2026-10-01
|
|
# 2 x 262,144 shape was chosen to hold a 216,269-token compaction request;
|
|
# three contexts is the trade Frank asked for).
|
|
CTX_SIZE=589824
|
|
PARALLEL=3
|
|
|
|
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
|
# q8_0 on V is only served by the flash-attention path; the serve script
|
|
# pins --flash-attn on and must never lose it.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|