serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF	The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves.	llm frank-small dankclaw llama-cpp rtx3090 mimo gguf kv-cache	2026-10-02

frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

ACTIVE since 2026-10-02. One quant step down from frank-small-llama-mimo-q6k on the same runtime, the same shape and the same KV contract. Frank's order: bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on llama.cpp, two contexts, whatever context size fits the 3090. The answer to "whatever fits" is unchanged — 2 x 262,144, because 262,144 is the model's native window and two of them already fit.

Identity

Field	Value
Endpoint	192.168.0.143:7771, served alias frank-small
Runtime	llama.cpp master (qwen35 model support), CUDA build sm_86, at /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server — unchanged across the quant swap
Payload	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M, 6,876,124,704 B (6.40 GiB), sha256 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f, at /home/dank/models/MiMo-Q5KM/ (fetched by scripts/download-frank-small.sh)
Unit	root frank-small.service (repo scripts/frank-small.service, User=dank, relinked), env scripts/frank-small.env, serve scripts/serve-frank-small.sh
Repo pins	tests/frank_small_unit.bats (stub-argv seam + the VRAM arithmetic), deploy_llm_small gates in scripts/deploy-remote.sh

Shape and the KV contract

--ctx-size 524288 --parallel 2 — llama.cpp divides the pool across slots, so each session gets n_ctx_slot = 262144, the model's full native window (boot log confirms n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false').

The KV contract is Frank's: K unquantized (f16), V q8_0 — --cache-type-k f16 --cache-type-v q8_0. Two hard dependencies ride on it:

• --flash-attn on is mandatory: quantized V exists only on the FA path. Without the flag the engine falls back silently and the contract becomes a lie. The serve script pins the trio; the bats seam pins the script.
• The VRAM budget: attention KV lives in the 8 full-attention layers only (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈ 12.5 GiB. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) + compute buffers/CUDA graphs ⇒ 20,164 MiB measured against the card's 24,576 MiB.

Why the quant step did not buy a bigger window. The model cannot be run past its trained 262,144 without scaling, and two of those windows are already the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That arithmetic is a test (the KV budget fits the 3090…), so a future CTX_SIZE or quant change that breaks the fit fails the suite instead of the boot.

Measured on this file (2026-10-02)

Check	Result
Boot to healthy	~20 s, n_slots = 2, n_ctx_slot = 262144
VRAM	20,164 MiB / 24,576 MiB (nvidia-smi)
Decode, temp 0, no thinking	100.4–101.8 tok/s (256–512 token completions at ~900-token depth)
Two concurrent requests	both answered, 256 tokens each in 3.2 s wall (slots are real)
Tools	native tool_calls with correct arguments (get_weather {"city":"Seattle","unit":"c"})
Thinking	reasoning_content populated on a reasoning-requiring prompt

Client contract

• Thinking arrives in reasoning_content (--reasoning-format deepseek), the field models.yml reads for this endpoint.
• Thinking is off unless the request asks: the MiMo GGUF template defaults enable_thinking false, so models.yml ships chat_template_kwargs: {enable_thinking: true, preserve_thinking: true} in extraBody. A raw client that omits it gets a direct answer — measured, and not a fault. Trivial questions produce no thinking even with the flag on.
• Tools: --jinja + server-side parsing, verified 2026-10-02.
• Text-only. This GGUF carries no vision tensors. The HF repo does ship a paired mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf, so vision is one --mmproj away if it is ever wanted here (frank-primary is the fleet's vision endpoint today).

No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00 ghost), and the cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf) is an optimization candidate, not a default — acceptance has to be measured on this traffic first. The bats seam fails if a draft flag ships unannounced.

Rollback

Target	How
Q6_K, same runtime	scripts/frank-small.env MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf (7,793,710,624 B, still on disk) and the deployer's LLM_SMALL_MODEL_FILE default
SGLang + INT8	frank-small-sglang-mimo-v26 — venv and payload both remain on disk; the fetch recipe moved to old/download-frank-small-sglang-int8.sh

When it misbehaves

• Port 7771 dark: journalctl -u frank-small -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved.
• No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing — nvidia-modprobe -u -c=0 (the serve script runs it).
• Requests queue behind each other: two concurrent long generations are the slot ceiling by contract.
• Slow decode after a CTX or quant change: check nvidia-smi — pushing KV out of VRAM into unified memory halves the rate silently.