Frank
6584c059d6
Frank's order, two messages: "extend the max context of frank-small to 200700 per individual stream without further quantization", then "reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to 1k", then "unquantize the KV if we have extra space". CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704: above the 200,700 the harness compacts at, so a full-length request lands in one stream instead of a 400 exceed_context_size_error. The pool must divide evenly across slots and stay 256-aligned for the FA path, which is why it is 401,408 and not 401,400. CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for unquantized values (24.5 -> 32 KiB/token), and the budget lands back in the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of the card idle. The asymmetric-KV contract that created this endpoint is spent with it; both --cache-type flags stay explicit. --batch-size/--ubatch-size are now passed (2048/1024). The endpoint had never passed either, so it ran on engine defaults nobody wrote down. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
79 lines
4.2 KiB
Bash
79 lines
4.2 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
|
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
|
# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
|
|
# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
|
|
# up is exactly the VRAM that pays for unquantized V. No further quantization
|
|
# anywhere on this endpoint.
|
|
#
|
|
# Three measurements on this card, this runtime, price the terms:
|
|
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
|
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
|
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
|
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
|
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
|
# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
|
|
# shape) => non-KV = 8,748 MiB
|
|
# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
|
|
# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
|
|
# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
|
|
# shape hit, bought with precision instead of a slot. The 1024 ubatch
|
|
# re-prices the compute buffers upward, so MEASURE after any change: if the
|
|
# boot lands under 2 GiB free, step the pool down in 12,288-token
|
|
# (384 MiB at this rate) increments.
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
|
|
# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
|
|
# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
|
|
# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
|
|
# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
|
|
# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
|
|
# precision buys; q8_0 on V would buy 262,144 instead.
|
|
#
|
|
# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
|
|
# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
|
|
# and the third slot is gone. The flags stay explicit so the choice is visible
|
|
# in the argv either way.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
|
|
|
# Vision: the Q5_K_M GGUF ships no `v.*` tensors, so the projector file IS the
|
|
# vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/MiMo-Q5KM/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 200,704 per slot is BELOW the native 262,144 — the price of paying for the
|
|
# window in full precision — and ABOVE the harness's compaction trigger, which
|
|
# is the point: a full-length request now lands in one stream instead of a 400
|
|
# exceed_context_size_error. Cost of the third slot going away: a third
|
|
# concurrent request queues behind a released slot instead of running.
|
|
CTX_SIZE=401408
|
|
PARALLEL=2
|
|
|
|
# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
|
|
# so it ran on llama.cpp's own defaults (2048/512); the fleet now asks for a
|
|
# 1024 ubatch on this shape. Both are tunables because the pair is a
|
|
# measurement, not a law: on the 27B sibling 512 was the prefill peak and every
|
|
# step up cost GiB of compute buffer for no speed.
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=1024
|
|
|
|
# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays
|
|
# pinned on in the serve script — it is what made the q8_0-V path cheap and it
|
|
# remains the fast path for f16 KV.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
NGL=99
|