Frank
d82c5c03b7
One quant step down from Q6_K on the same CUDA build and the same shape: 2 x 262,144 with K f16 + V q8_0. The 0.86 GiB freed from the weights is headroom, not a new window — the native 262,144 cap is already reached, and f16 V would still cost 3.96 GiB more than the pool can carry.
44 lines
2.1 KiB
Bash
44 lines
2.1 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-01 night (Frank): one quant step down from the Q6_K file this
|
|
# endpoint ran for a few hours, on the same runtime and the same shape — the
|
|
# llama.cpp GGUF path is kept for the ASYMMETRIC KV contract (K unquantized
|
|
# f16, V q8_0) that no other runtime on this host exposes per-tensor, and the
|
|
# freed 0.86 GiB of weights is what keeps the two full windows bootable with
|
|
# margin for CUDA graphs.
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token; V q8_0 = 8.5 KiB/token
|
|
# (34 B per 32-element block) => 24.5 KiB/token. Two full native windows
|
|
# (2 x 262,144 = 524,288 tokens) cost 12.5 GiB; weights 6.40 + KV 12.5 + the
|
|
# fixed Gated DeltaNet state (~25 MiB/slot) + compute buffers and CUDA graphs
|
|
# ~1.5-2 => ~20.5 GiB against the card's 24 GiB. Q6_K measured 20.9/24.6 here,
|
|
# which is the same arithmetic with 7.26 GiB of weights.
|
|
#
|
|
# Why V is not f16: full-precision V costs 16 KiB/token instead of 8.5, i.e.
|
|
# +3.96 GiB on a 524,288-token pool — 24.5 GiB total, past the card. If the
|
|
# graph capture OOMs, lower CTX_SIZE (262144 = two 131,072 windows) rather
|
|
# than quantizing K.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two contexts at the model's native window each. llama.cpp divides
|
|
# --ctx-size across --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and
|
|
# without unified KV each slot gets its own cache.
|
|
CTX_SIZE=524288
|
|
PARALLEL=2
|
|
|
|
# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V.
|
|
# q8_0 on V is only served by the flash-attention path; the serve script
|
|
# pins --flash-attn on and must never lose it.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|