serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF	The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves.	llm frank-small dankclaw llama-cpp rtx3090 mimo gguf kv-cache	2026-10-02

frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

ACTIVE since 2026-10-02. One quant step down from frank-small-llama-mimo-q6k on the same runtime and the same KV contract, then resized twice the same day on Frank's orders: first to three contexts with vision, then down to the pool that keeps 2–3 GiB of the card free. Frank's arithmetic request — "as much as can fit while keeping overhead open space to 2-3gb on the 3090" — is answered by the two measurements in Shape and the KV contract below; the answer is 3 × 180,224, not 3 × 262,144.

The tradeoff is the per-slot window. Three sessions cost 81,920 tokens of per-conversation memory versus the two-slot shape: 180,224 tokens each instead of the model's native 262,144. See What three slots cost.

Identity

Field	Value
Endpoint	192.168.0.143:7771, served alias frank-small
Runtime	llama.cpp master (qwen35 model support), CUDA build sm_86, at /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server — unchanged across the quant swap
Payload	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M, 6,876,124,704 B (6.40 GiB), sha256 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f + the repo's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf, 918,166,048 B, sha256 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576 — both at /home/dank/models/MiMo-Q5KM/, fetched and byte- and sha256-verified by scripts/download-frank-small.sh
Unit	root frank-small.service (repo scripts/frank-small.service, User=dank, relinked), env scripts/frank-small.env, serve scripts/serve-frank-small.sh
Repo pins	tests/frank_small_unit.bats (stub-argv seam + the VRAM arithmetic), deploy_llm_small gates in scripts/deploy-remote.sh

Shape and the KV contract

--ctx-size 540672 --parallel 3 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 180224 (boot log: n_slots = 3, n_ctx_slot = 180224, kv_unified = 'false'). CTX_SIZE MUST stay divisible by PARALLEL; the bats seam enforces it.

The KV contract is Frank's: K unquantized (f16), V q8_0 — --cache-type-k f16 --cache-type-v q8_0. Two hard dependencies ride on it:

• --flash-attn on is mandatory: quantized V exists only on the FA path. Without the flag the engine falls back silently and the contract becomes a lie. The serve script pins the trio; the bats seam pins the script.
• The VRAM budget: attention KV lives in the 8 full-attention layers only (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.

The pool size came from two boots, not from a formula. free-style arithmetic on this stack is unreliable because the compute buffers, CUDA graphs and now the vision tower all bill separately, so the fit was measured at two shapes and solved between them:

Shape	Projector	nvidia-smi used	KV at 24.5 KiB/token	Non-KV (weights + buffers + projector)
2 × 262,144 (pool 524,288)	none	20,190 MiB	12,800 MiB	7,390 MiB
3 × 196,608 (pool 589,824)	f16	22,972 MiB	14,113 MiB	8,859 MiB

The two rows price the projector and its image buffers at ~1.4 GiB — not the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own activation and compute buffers ride along. Solving the second row's non-KV cost against Frank's 2–3 GiB band: the card is 24,576 MiB, so KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB ⇒ pool ≤ 556,630 tokens, rounded down to a multiple of 3 × 4,096 ⇒ 540,672, i.e. 13,265 MiB of KV and 2,441 MiB free (measured, 21,684 MiB used).

Why 3 × 262,144 was never possible. Three native windows are 786,432 tokens = 18.4 GiB of KV alone; plus the 8.86 GiB of non-KV cost that is 27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed 6 GiB.

Measured on this shape (2026-10-02, 3 × 180,224 + vision)

Check	Result
Boot to healthy	~20 s, n_slots = 3, n_ctx_slot = 180224, loaded multimodal model logged
VRAM	21,684 MiB used / 2,441 MiB free of 24,576 (nvidia-smi, steady state; the KV pool is allocated whole at boot, so this is the ceiling)
Decode, temp 0, no thinking	106.8 tok/s (256-token completion at ~1,300-token depth)
Prefill	3,814 tok/s (same request)
Three concurrent requests	all three answered in 1.7 s wall — real parallel slots
Fourth concurrent request	answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires
Vision	image_url part with a 64×64 solid-red PNG → "Red."
Tools	native tool_calls with correct arguments (get_weather {"city":"Seattle","unit":"f"})
Thinking	reasoning_content populated (145 chars on a "think first" prompt); reasoning_effort: "low" accepted
Over-slot prompt	400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens) — refused, never silently truncated

Decode is ~5 % faster than the 2 × 262,144 shape measured earlier the same day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention scan at equal prompt depth, and one less slot's DeltaNet state resident.

What three slots cost

The per-slot window is the whole price. Two consequences the fleet depends on:

• A prompt over 180,224 tokens is a hard 400, so ~/.omp/agent/models.yml sets contextWindow: 180224 for this endpoint — the harness compacts before it reaches the ceiling instead of discovering it.
• frank-primary's compaction prompt fits inside one slot again. At frank-primary's shape-D window (200,704) its compaction trigger is 140,492 tokens (70 %), so one compaction request lands here with 39,732 tokens of room. At the previous 262,144 window the trigger was 183,501 — above this slot — and the request had to be truncated or split. Nothing in the fleet routes compaction here automatically; the point is that the 2026-10-02 window trim restored the fit.

Client contract

• Thinking arrives in reasoning_content (--reasoning-format deepseek), the field models.yml reads for this endpoint.
• Thinking is off unless the request asks: the MiMo GGUF template defaults enable_thinking false, so models.yml ships chat_template_kwargs: {enable_thinking: true, preserve_thinking: true} in extraBody. A raw client that omits it gets a direct answer — measured, and not a fault. Trivial questions produce no thinking even with the flag on.
• Tools: --jinja + server-side parsing, verified 2026-10-02.
• Vision is on, through the projector file (--mmproj). Use the OpenAI-standard image_url content part — that path answers correctly ("Red." on a solid-red PNG). The legacy top-level images field is not read by /v1/chat/completions: the same request sent that way answered "Brown.", i.e. it generated from text alone. /completion takes images but refuses a request without messages in this build, so chat is the only vision path.
• Setting MMPROJ_FILE= (explicitly empty) in scripts/frank-small.env drops --mmproj and returns ~1.4 GiB of the card to KV — the documented text-only opt-out. MMPROJ_FILE pointing at a file that does not exist is a boot refusal, never a silent fallback to blindness.

No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00 ghost), and the cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf) is an optimization candidate, not a default — acceptance has to be measured on this traffic first. The bats seam fails if a draft flag ships unannounced.

Rollback

Target	How
Q6_K, same runtime	scripts/frank-small.env MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf (7,793,710,624 B, still on disk) and the deployer's LLM_SMALL_MODEL_FILE default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with CTX_SIZE 49,152 tokens lower or MMPROJ_FILE=
Two 262,144 windows	CTX_SIZE=524288 PARALLEL=2 (measured 20,190 MiB used, 4,386 free) and contextWindow: 262144 in models.yml — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here
Vision off	MMPROJ_FILE= in the env file; the deployer gate then has to be satisfied too, so point LLM_SMALL_MMPROJ at any real projector file or drop the gate in the same commit
SGLang + INT8	frank-small-sglang-mimo-v26 — venv and old unit/env/serve trio still in place; the INT8 payload was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ on 2026-10-02, so restore it there or re-fetch with old/download-frank-small-sglang-int8.sh

When it misbehaves

• Port 7771 dark: journalctl -u frank-small -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved.
• No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing — nvidia-modprobe -u -c=0 (the serve script runs it).
• Requests queue behind each other: three concurrent long generations are the slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s for the three, 3.0 s for the fourth).
• 400 … exceeds the available context size (180224 tokens): the conversation outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens and the per-slot window is the binding limit. Shorten the prompt, compact, or roll back to the 262,144 shape above.
• Vision answers a color it never saw: the client sent images instead of an image_url content part (see Client contract).
• Slow decode after a CTX or quant change: check nvidia-smi — pushing KV out of VRAM into unified memory halves the rate silently.