serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV	The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change.	llm frank-small dankclaw llama-cpp gguf rtx3090	2026-10-03

frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV

ACTIVE since 2026-10-03. The third shape Frank bought in two days on this card: the freed third slot paid for exact KV values (frank-small-llama-mimo-q5km documents that step and remains the rollback), and the ~3.5 GiB still free then paid for the Q6_K quant rung — the same file this endpoint served on 2026-10-01, come back as an upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704); only the weights moved up.

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	http://192.168.0.143:7771 — alias frank-small, /v1 OpenAI surface
Runtime	llama.cpp master, CUDA sm_86 build …/llama-master/build-cuda/bin/llama-server (GGML_CUDA_FA_ALL_QUANTS=ON)
Unit	root systemd frank-small.service (scripts/frank-small.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-small.sh, tunables scripts/frank-small.env
Weights	/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K, 7,793,710,624 B, sha256 ef96d05a… (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors)
Projector	…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf — 918,166,048 B, sha256 ff348f… (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained)
Fetch	scripts/download-frank-small.sh — -f, byte count + LFS sha256 verified, --verify-only; both files verified against the pinned sha on the host before this deploy
Seam	tests/frank_small_unit.bats (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out)

Shape and KV contract

--ctx-size 401408 --parallel 2 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 200704. CTX_SIZE MUST stay divisible by PARALLEL; the seam enforces it. Both K and V are unquantized (f16) — the flags stay explicit because --flash-attn must not be lost if V ever goes back to q8_0 (the serve script pins it, the seam pins the pin).

The Q6_K rung is a weights-side spend only. File delta +917,585,920 B (+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair. Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot landed 21,276 MiB used / 2,849 MiB true free — non-KV measured 8,732 MiB, inside the 2–3 GiB floor, so the pool stayed at 401,408 and the 12,288-token step-down ladder was never climbed. Quote nvidia-smi's free field, never total − used: the driver holds ~450 MiB of reserve the subtraction hides (that error once overstated this card's slack by a half-GiB).

Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here). Q8_0 is the only rung above this one on the Hub (9,545,979,424 B = 9,104 MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free — under the floor — and the calibrated bandwidth law (it predicted this file's 94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further −24 % for the near-lossless rung. Fitting it was priced two ways (trim the pool to 2 × 182,272 — which also breaks frank-large's 183,501-token compaction-prompt fit by 1,236 tokens — or revert V to q8_0), and both were declined. Re-open only on evidence the Q6_K residual matters.

Pool prices on this card come from boots, never formulas — the measured boot-pair table lives in frank-small-llama-mimo-q5km (Shape and KV contract) and still prices every rollback shape.

Compute-buffer geometry (2026-10-05)

BATCH_SIZE=4096 UBATCH_SIZE=2048 — both doubled from 2048/1024, and the trade measured rather than assumed, because the predecessor shape recorded "ubatch 2048 failed the scratch boot" and this card has less free space than it did.

That failure was a different shape, and it did not reproduce here. The service itself is the bench (one card holds one server, so no scratch port): the old rung was re-booted through a transient FRANK_SMALL_ENV_FILE drop-in and reproduced the 2026-10-03 boot byte-identically — 21,276 MiB used / 2,849 MiB free — which makes the ubatch the only variable in the comparison.

Probe	ubatch 1024	ubatch 2048	Δ
Prefill, 8,928 tokens	3,972 / 4,077 t/s	4,207 / 4,212 t/s	+4.6 %
Decode, 128 tokens at that depth	91.87 / 93.01 t/s	93.81 / 94.27 t/s	noise
Prefill, 279 tokens	2,675.7 t/s	2,658–2,811 t/s	noise
VRAM used / free	21,276 / 2,849 MiB	21,846 / 2,279 MiB	−616 MiB

Three readings:

• The gain only exists above the old ubatch. A 279-token prompt is shorter than either micro-batch, so it cannot see the knob — and it did not move. That is the control proving the 8.9k delta is geometry, not drift.
• Decode is flat, as every prior sweep on this card found. Ubatch is a prefill knob.
• The price is 616 MiB out of a 2.8 GiB margin. 2,279 MiB free is still inside the 2–3 GiB band, at its bottom edge. 4096/1536 is the middle rung, and the pool's 12,288-token (384 MiB) ladder in frank-small-llama-mimo-q5km is the other way to buy room back.

Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is backend-specific. The 2× R9700 layer split peaks at ubatch 512 and is 10 % off that peak by 2048 (frank-medium-batch-ubatch); on this CUDA card the curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep, 1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a ubatch verdict across backends.

The logical --batch-size is VRAM- and throughput-neutral — two sweeps and this A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because that is the pair the curve was measured at.

Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)

Check	Result
Boot to healthy	model + projector resident in ~2.3 s (load log 0.02.27), n_slots = 2, n_ctx_slot = 200704, loaded multimodal model, health 200; deploy e2e gate 33/33
VRAM	21,276 MiB used / 2,849 MiB free of 24,576 (nvidia-smi free, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: 21,846 / 2,279 — see Compute-buffer geometry
Decode, temp 0, no thinking	94.44 tok/s on the identical 506-token probe that measured 104.84 on Q5_K_M — −10 %. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably
Prefill	2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)
Vision	512×512 solid-red PNG → Red (277 prompt tokens — the image really reached the tower). See the probe-size trap in Client contract
Tools / thinking	native tool_calls, thinking in reasoning_content, thinking OFF unless enable_thinking: true (MiMo template default; models.yml ships it in extraBody)

Client contract

llama.cpp surface, live-verified on this model family: model id frank-small, native tools via --jinja (tool_calls in the response), thinking in reasoning_content, vision via image_url content parts. Two vision traps, both measured here:

1. A top-level images field on /v1/chat/completions is silently ignored and the model invents a color.
2. The probe image must be big enough to supply real image tokens. A 10×10 px solid-red PNG came back "The image is a video, so I can't see its content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image tokens, and a postage-stamp image lands one patch below usable signal. 512×512 answers Red in 19 tokens. Always probe vision with a ≥ 256 px image; a wrong-color answer from a tiny image is the probe's fault, not the endpoint's.

~/.omp/agent/models.yml sets contextWindow: 200704 — the PER-SLOT window, so the harness compacts before llama.cpp answers a longer prompt with a 400.

Rollback

Target	How
Q5_K_M at the same shape	scripts/frank-small.env MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf (+ MMPROJ_FILE in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) and the 10 % decode rate (104.84 tok/s measured). Payload sha-verified on disk
3 × 180,224 + q8_0 V	CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0 (measured 21,684 used / 2,441 free with Q5_K_M — Q6_K does NOT fit that shape inside the floor; pair them explicitly)
SGLang + INT8	frank-small-sglang-mimo-v26 — payload staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/

When it misbehaves

Same decision tree as frank-small-llama-mimo-q5km (When it misbehaves): No CUDA GPUs are available ⇒ nvidia-modprobe -u -c=0; a 400 naming 200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after a shape change ⇒ check nvidia-smi for KV pushed into unified memory; queue behind two long generations ⇒ the slot ceiling is two, by contract.