serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md

---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
tags:
  - llm
  - frank-small
  - dankclaw
  - llama-cpp
  - gguf
  - rtx3090
timestamp: 2026-10-03
---

# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV

> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
> this card: the freed third slot paid for **exact KV values**
> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
> — the same file this endpoint served on 2026-10-01, come back as an
> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
> only the weights moved up.

## Identity

| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |

## Shape and KV contract

`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).

**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
field, never `total − used`: the driver holds ~450 MiB of reserve the
subtraction hides (that error once overstated this card's slack by a
half-GiB).

**Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here).**
`Q8_0` is the only rung above this one on the Hub (9,545,979,424 B = 9,104
MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
under the floor — and the calibrated bandwidth law (it predicted this file's
94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
pool to 2 × 182,272 — which also breaks frank-large's 183,501-token
compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
were declined. Re-open only on evidence the Q6_K residual matters.

**Pool prices on this card come from boots, never formulas** — the measured
boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
contract*) and still prices every rollback shape.

## Compute-buffer geometry (2026-10-05)

`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
measured rather than assumed, because the predecessor shape recorded **"ubatch
2048 failed the scratch boot"** and this card has less free space than it did.

**That failure was a different shape, and it did not reproduce here.** The
service itself is the bench (one card holds one server, so no scratch port):
the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
2,849 MiB free — which makes the ubatch the only variable in the comparison.

|Probe|ubatch 1024|ubatch 2048|Δ|
|---|---|---|---|
|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|

Three readings:

- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
  than either micro-batch, so it cannot see the knob — and it did not move.
  That is the control proving the 8.9k delta is geometry, not drift.
- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
  prefill knob.
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
  inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
  and the pool's 12,288-token (384 MiB) ladder in
  [[frank-small-llama-mimo-q5km]] is the other way to buy room back.

**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
ubatch verdict across backends.

The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
that is the pair the curve was measured at.

## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)

|Check|Result|
|---|---|
|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|

## Client contract

llama.cpp surface, live-verified on this model family: model id `frank-small`,
native tools via `--jinja` (`tool_calls` in the response), thinking in
**`reasoning_content`**, vision via `image_url` content parts. Two vision
traps, both measured here:

1. A top-level `images` field on `/v1/chat/completions` is silently ignored
   and the model invents a color.
2. **The probe image must be big enough to supply real image tokens.** A
   10×10 px solid-red PNG came back "The image is a video, so I can't see its
   content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
   tokens, and a postage-stamp image lands one patch below usable signal.
   512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
   image; a wrong-color answer from a tiny image is the probe's fault, not
   the endpoint's.

`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
so the harness compacts before llama.cpp answers a longer prompt with a 400.

## Rollback

|Target|How|
|---|---|
|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|

## When it misbehaves

Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
behind two long generations ⇒ the slot ceiling is two, by contract.