Frank
934285c2b2
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
61 lines
3.2 KiB
Bash
61 lines
3.2 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to
|
|
# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot
|
|
# window shrinks — see the budget below.
|
|
#
|
|
# Two measurements on this card, this runtime, price the terms:
|
|
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
|
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
|
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
|
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
|
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
|
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
|
|
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
|
|
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
|
|
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
|
|
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
|
|
# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot)
|
|
# is one 293 MiB step — that is the ladder to walk if this moves again. The
|
|
# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x
|
|
# 24.5 KiB = 18.4 GiB of KV before anything else is loaded.
|
|
#
|
|
# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token —
|
|
# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is
|
|
# the tensor the asymmetric-KV contract exists to protect.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
|
|
|
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
|
|
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Three independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 180,224 per slot is BELOW the native 262,144, and that is the cost of the
|
|
# third slot: a longer single prompt needs compaction instead of one shot.
|
|
# Consequence worth knowing: frank-primary's compaction trigger is 183,501
|
|
# tokens, which no longer fits a frank-small slot — see the runbook's
|
|
# "three slots vs compaction" note.
|
|
CTX_SIZE=540672
|
|
PARALLEL=3
|
|
|
|
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
|
# q8_0 on V is only served by the flash-attention path; the serve script
|
|
# pins --flash-attn on and must never lose it.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|