serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF	The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves.	llm frank-small dankclaw llama-cpp rtx3090 mimo gguf kv-cache	2026-10-03

frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

ACTIVE since 2026-10-02, resliced 2026-10-03. One quant step down from frank-small-llama-mimo-q6k on the same runtime, resized twice on 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of the card free), and on 2026-10-03 Frank ordered the third shape: two conversations of 200,704 tokens each with the KV VALUES unquantized — "2 streams with 200700 tokens context or more… if we have extra space on the 3090, unquantize the kv." The freed third slot pays for exact V, and the boot lands 20,530 MiB used / 4,046 MiB free — above the 2–3 GiB floor, with full-precision KV.

The tradeoffs of the reslice: one fewer simultaneous conversation, a per-slot window that grew from 180,224 to 200,704 (still below the native 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger per-token KV traffic. The 3 × 180,224 shape is one env edit away — see Rollback.

Identity

Field	Value
Endpoint	192.168.0.143:7771, served alias frank-small
Runtime	llama.cpp master (qwen35 model support), CUDA build sm_86, at /home/dank/GitPackages/llama-master/build-cuda/bin/llama-server — unchanged across the quant swap
Payload	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M, 6,876,124,704 B (6.40 GiB), sha256 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f + the repo's mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf, 918,166,048 B, sha256 ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576 — both at /home/dank/models/MiMo-Q5KM/, fetched and byte- and sha256-verified by scripts/download-frank-small.sh
Unit	root frank-small.service (repo scripts/frank-small.service, User=dank, relinked), env scripts/frank-small.env, serve scripts/serve-frank-small.sh
Repo pins	tests/frank_small_unit.bats (stub-argv seam + the VRAM arithmetic), deploy_llm_small gates in scripts/deploy-remote.sh

Shape and the KV contract

--ctx-size 401408 --parallel 2 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 200704 (boot log 2026-10-03: n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'). CTX_SIZE MUST stay divisible by PARALLEL; the bats seam enforces it.

The KV contract is Frank's, as of 2026-10-03: both K and V unquantized (f16) — --cache-type-k f16 --cache-type-v f16. The asymmetric contract it replaced (f16 K + q8_0 V) had existed to make three slots fit; with two slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra V traffic is the measured ~2 % decode tax. The serve script still passes both flags explicitly — the asymmetric-era law that a quantized V requires --flash-attn on stays pinned by the seam.

Pool sizes on this stack come from boots, not formulas. The compute buffers, CUDA graphs and vision tower all bill separately, so every shape change is priced from measured rows:

Shape	KV dtype (K+V)	Projector	nvidia-smi used	Non-KV (weights + buffers + projector)
2 × 262,144 (pool 524,288)	f16 + q8_0	none	20,190 MiB	7,390 MiB
3 × 196,608 (pool 589,824)	f16 + q8_0	f16	22,972 MiB	8,859 MiB
3 × 180,224 (pool 540,672)	f16 + q8_0	f16	21,684 MiB	~8,419 MiB
2 × 200,704 (pool 401,408)	f16 + f16	f16	20,530 MiB	~7,986 MiB

The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim × 2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = 12,544 MiB of KV; the card is 24,576 MiB, so the shape leaves 4,046 MiB free (measured steady-state — the KV pool is allocated whole at boot, so this is the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not an invitation: another 32,768 tokens of pool (1 GiB) would trade the last safety margin for window the harness compacts before reaching anyway.

Why 2 × 262,144 with f16 V does not fit either. The native window pair is 524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the measured band.

Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)

Check	Result
Boot to healthy	~5 s, n_slots = 2, n_ctx_slot = 200704, loaded multimodal model logged
VRAM	20,530 MiB used / 4,046 MiB free of 24,576 (nvidia-smi, steady state)
Decode, temp 0, no thinking	104.84 tok/s (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)
Prefill	1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)
Batch geometry	--batch-size 2048 --ubatch-size 1024 — ubatch 2048 failed the scratch boot; 1024 boots clean
Vision	projector loads at boot (loaded multimodal model); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice
Tools / thinking	unchanged by the reslice: native tool_calls, thinking in reasoning_content

What the slot geometry costs

The per-slot window is the whole price, in both directions:

• A prompt over 200,704 tokens is a hard 400, so ~/.omp/agent/models.yml sets contextWindow: 200704 for this endpoint — the harness compacts before it reaches the ceiling instead of discovering it.
• frank-large's tallest pre-compaction prompt now fits here. Compaction for frank-large sessions runs on this endpoint (compactionModel in models.yml); frank-large compacts at 70 % of 262,144 = 183,501 tokens, which did not fit the old 180,224 slot and fits 200,704 with 17,163 tokens to spare. The 2026-10-02 window trim had restored the fit for frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens the fit to frank-large itself. Nothing routes compaction automatically — the point is the ceiling no longer forbids it.
• Two simultaneous generations are the slot ceiling; a third request queues behind a released slot (the mechanism measured on the 3-slot shape).

Client contract

• Thinking arrives in reasoning_content (--reasoning-format deepseek), the field models.yml reads for this endpoint.
• Thinking is off unless the request asks: the MiMo GGUF template defaults enable_thinking false, so models.yml ships chat_template_kwargs: {enable_thinking: true, preserve_thinking: true} in extraBody. A raw client that omits it gets a direct answer — measured, and not a fault. Trivial questions produce no thinking even with the flag on.
• Tools: --jinja + server-side parsing, verified 2026-10-02.
• Vision is on, through the projector file (--mmproj). Use the OpenAI-standard image_url content part — that path answers correctly ("Red." on a solid-red PNG). The legacy top-level images field is not read by /v1/chat/completions: the same request sent that way answered "Brown.", i.e. it generated from text alone. /completion takes images but refuses a request without messages in this build, so chat is the only vision path.
• Setting MMPROJ_FILE= (explicitly empty) in scripts/frank-small.env drops --mmproj and returns ~1.4 GiB of the card to KV — the documented text-only opt-out. MMPROJ_FILE pointing at a file that does not exist is a boot refusal, never a silent fallback to blindness.

No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00 ghost), and the cross-model DFlash draft (/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf) is an optimization candidate, not a default — acceptance has to be measured on this traffic first. The bats seam fails if a draft flag ships unannounced.

Rollback

Target	How
Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)	scripts/frank-small.env CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0 (measured 21,684 MiB used, 2,441 free) and contextWindow: 180224 in models.yml — choose it when three simultaneous conversations matter more than per-slot window or exact V
Q6_K, same runtime	scripts/frank-small.env MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf (7,793,710,624 B, still on disk) and the deployer's LLM_SMALL_MODEL_FILE default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with CTX_SIZE 49,152 tokens lower or MMPROJ_FILE=
Two 262,144 windows	CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0 (measured 20,190 MiB used, 4,386 free with q8_0 V — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and contextWindow: 262144 in models.yml — choose it when a single conversation needs the native window
Vision off	MMPROJ_FILE= in the env file; the deployer gate then has to be satisfied too, so point LLM_SMALL_MMPROJ at any real projector file or drop the gate in the same commit
SGLang + INT8	frank-small-sglang-mimo-v26 — venv and old unit/env/serve trio still in place; the INT8 payload was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ on 2026-10-02, so restore it there or re-fetch with old/download-frank-small-sglang-int8.sh

When it misbehaves

• Port 7771 dark: journalctl -u frank-small -n 50; the serve script exits 1 naming the missing file if the binary or GGUF moved.
• No CUDA GPUs are available: the 3090 vanished from the bus or the UVM node is missing — nvidia-modprobe -u -c=0 (the serve script runs it).
• Requests queue behind each other: two concurrent long generations are the slot ceiling by contract (2026-10-03 shape); a third waits for a slot to free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for the fourth).
• 400 … exceeds the available context size (200704 tokens): the conversation outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens and the per-slot window is the binding limit. Shorten the prompt, compact, or roll back to the 262,144 shape above.
• Vision answers a color it never saw: the client sent images instead of an image_url content part (see Client contract).
• Slow decode after a CTX or quant change: check nvidia-smi — pushing KV out of VRAM into unified memory halves the rate silently.