Frank
7fe881de01
Measured after the first 3-slot deploy: 3 x 196,608 booted (n_slots=3, n_ctx_slot=196608, multimodal loaded) but used 22,972/24,576 MiB - 1,153 MiB free, outside the 2-3 GiB band. The two measurements now recorded in frank-small.env price the vision tower at ~1.4 GiB (non-KV 7,390 -> 8,859 MiB), so the pool drops 49,152 tokens (16,384/slot, 1,176 MiB) to land ~2.7 GiB free. The bats budget gate is recalibrated to the measured 9 GiB non-KV cost, which now rejects the 3 x 196,608 shape instead of blessing it.
60 lines
3.1 KiB
Bash
60 lines
3.1 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
|
|
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
|
|
# window shrinks — see the budget below.
|
|
#
|
|
# Two measurements on this card, this runtime, price the terms:
|
|
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
|
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
|
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
|
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
|
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
|
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
|
|
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
|
|
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB used,
|
|
# ~2.7 GiB free, the middle of what was asked for.
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
|
|
# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
|
|
# is one 293 MiB step — that is the ladder to walk if this moves again. The
|
|
# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
|
|
# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
|
|
#
|
|
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
|
|
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
|
|
# the tensor the asymmetric-KV contract exists to protect.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
|
|
|
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
|
|
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Three independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
|
|
# third slot: a longer single prompt needs compaction instead of one shot.
|
|
# Consequence worth knowing: frank-primary's compaction trigger is 183,501
|
|
# tokens, which no longer fits a frank-small slot — see the runbook's
|
|
# "three slots vs compaction" note.
|
|
CTX_SIZE=540672
|
|
PARALLEL=3
|
|
|
|
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
|
# q8_0 on V is only served by the flash-attention path; the serve script
|
|
# pins --flash-attn on and must never lose it.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|