serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV	The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change.	llm frank-small dankclaw llama-cpp gguf rtx3090	2026-10-03

frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV

ACTIVE since 2026-10-03. The third shape Frank bought in two days on this card: the freed third slot paid for exact KV values (frank-small-llama-mimo-q5km documents that step and remains the rollback), and the ~3.5 GiB still free then paid for the Q6_K quant rung — the same file this endpoint served on 2026-10-01, come back as an upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704); only the weights moved up.

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	http://192.168.0.143:7771 — alias frank-small, /v1 OpenAI surface
Runtime	llama.cpp master, CUDA sm_86 build …/llama-master/build-cuda/bin/llama-server (GGML_CUDA_FA_ALL_QUANTS=ON)
Unit	root systemd frank-small.service (scripts/frank-small.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-small.sh, tunables scripts/frank-small.env
Weights	/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K, 7,793,710,624 B, sha256 ef96d05a… (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors)
Projector	…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf — 918,166,048 B, sha256 ff348f… (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained)
Fetch	scripts/download-frank-small.sh — -f, byte count + LFS sha256 verified, --verify-only; both files verified against the pinned sha on the host before this deploy
Seam	tests/frank_small_unit.bats (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out)

Shape and KV contract

--ctx-size 401408 --parallel 2 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 200704. CTX_SIZE MUST stay divisible by PARALLEL; the seam enforces it. Both K and V are unquantized (f16) — the flags stay explicit because --flash-attn must not be lost if V ever goes back to q8_0 (the serve script pins it, the seam pins the pin).

The Q6_K rung is a weights-side spend only. File delta +917,585,920 B (+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair. Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot landed 21,276 MiB used / 2,849 MiB true free — non-KV measured 8,732 MiB, inside the 2–3 GiB floor, so the pool stayed at 401,408 and the 12,288-token step-down ladder was never climbed. Quote nvidia-smi's free field, never total − used: the driver holds ~450 MiB of reserve the subtraction hides (that error once overstated this card's slack by a half-GiB).

Pool prices on this card come from boots, never formulas — the measured boot-pair table lives in frank-small-llama-mimo-q5km (Shape and KV contract) and still prices every rollback shape.

Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)

Check	Result
Boot to healthy	model + projector resident in ~2.3 s (load log 0.02.27), n_slots = 2, n_ctx_slot = 200704, loaded multimodal model, health 200; deploy e2e gate 33/33
VRAM	21,276 MiB used / 2,849 MiB free of 24,576 (nvidia-smi free, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung
Decode, temp 0, no thinking	94.44 tok/s on the identical 506-token probe that measured 104.84 on Q5_K_M — −10 %. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably
Prefill	2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)
Vision	512×512 solid-red PNG → Red (277 prompt tokens — the image really reached the tower). See the probe-size trap in Client contract
Tools / thinking	native tool_calls, thinking in reasoning_content, thinking OFF unless enable_thinking: true (MiMo template default; models.yml ships it in extraBody)

Client contract

llama.cpp surface, live-verified on this model family: model id frank-small, native tools via --jinja (tool_calls in the response), thinking in reasoning_content, vision via image_url content parts. Two vision traps, both measured here:

1. A top-level images field on /v1/chat/completions is silently ignored and the model invents a color.
2. The probe image must be big enough to supply real image tokens. A 10×10 px solid-red PNG came back "The image is a video, so I can't see its content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image tokens, and a postage-stamp image lands one patch below usable signal. 512×512 answers Red in 19 tokens. Always probe vision with a ≥ 256 px image; a wrong-color answer from a tiny image is the probe's fault, not the endpoint's.

~/.omp/agent/models.yml sets contextWindow: 200704 — the PER-SLOT window, so the harness compacts before llama.cpp answers a longer prompt with a 400.

Rollback

Target	How
Q5_K_M at the same shape	scripts/frank-small.env MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf (+ MMPROJ_FILE in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) and the 10 % decode rate (104.84 tok/s measured). Payload sha-verified on disk
3 × 180,224 + q8_0 V	CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0 (measured 21,684 used / 2,441 free with Q5_K_M — Q6_K does NOT fit that shape inside the floor; pair them explicitly)
SGLang + INT8	frank-small-sglang-mimo-v26 — payload staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/

When it misbehaves

Same decision tree as frank-small-llama-mimo-q5km (When it misbehaves): No CUDA GPUs are available ⇒ nvidia-modprobe -u -c=0; a 400 naming 200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after a shape change ⇒ check nvidia-smi for KV pushed into unified memory; queue behind two long generations ⇒ the slot ceiling is two, by contract.