---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
tags:
  - llm
  - frank-small
  - dankclaw
  - llama-cpp
  - rtx3090
  - mimo
  - gguf
  - kv-cache
timestamp: 2026-10-02
---

# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

> **ACTIVE since 2026-10-02.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
> contract, then **resized twice the same day** on Frank's orders: first to
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
> card free. Frank's arithmetic request — "as much as can fit while keeping
> overhead open space to 2-3gb on the 3090" — is answered by the two
> measurements in *Shape and the KV contract* below; the answer is
> **3 × 180,224**, not 3 × 262,144.

> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
> instead of the model's native 262,144. See *What three slots cost*.

## Identity

|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|

## Shape and the KV contract

`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
`PARALLEL`; the bats seam enforces it.

The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:

- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
  Without the flag the engine falls back silently and the contract becomes a
  lie. The serve script pins the trio; the bats seam pins the script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
  (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.

**The pool size came from two boots, not from a formula.** `free`-style
arithmetic on this stack is unreliable because the compute buffers, CUDA
graphs and now the vision tower all bill separately, so the fit was measured
at two shapes and solved between them:

|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|---|---|---|---|---|
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|

The two rows price the projector and its image buffers at **~1.4 GiB** — not
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
activation and compute buffers ride along. Solving the second row's non-KV cost
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
**2,441 MiB free** (measured, 21,684 MiB used).

**Why 3 × 262,144 was never possible.** Three native windows are 786,432
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
6 GiB.

## Measured on this shape (2026-10-02, 3 × 180,224 + vision)

|Check|Result|
|---|---|
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|Prefill|**3,814 tok/s** (same request)|
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|

Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
scan at equal prompt depth, and one less slot's DeltaNet state resident.

## What three slots cost

The per-slot window is the whole price. Two consequences the fleet depends on:

- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
  sets `contextWindow: 180224` for this endpoint — the harness compacts before
  it reaches the ceiling instead of discovering it.
- **frank-primary's compaction prompt fits inside one slot again.** At
  frank-primary's shape-D window (200,704) its compaction trigger is 140,492
  tokens (70 %), so one compaction request lands here with 39,732 tokens of
  room. At the previous 262,144 window the trigger was 183,501 — *above* this
  slot — and the request had to be truncated or split. Nothing in the fleet
  routes compaction here automatically; the point is that the 2026-10-02 window
  trim restored the fit.

## Client contract

- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
  the field `models.yml` reads for this endpoint.
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
  `enable_thinking` false, so `models.yml` ships
  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
  `extraBody`. A raw client that omits it gets a direct answer — measured, and
  not a fault. Trivial questions produce no thinking even with the flag on.
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
- **Vision is on**, through the projector file (`--mmproj`). Use the
  OpenAI-standard `image_url` content part — that path answers correctly
  ("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
  read by `/v1/chat/completions`: the same request sent that way answered
  "Brown.", i.e. it generated from text alone. `/completion` takes `images`
  but refuses a request without `messages` in this build, so chat is the only
  vision path.
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
  `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
  opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
  refusal, never a silent fallback to blindness.

## No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
ghost), and the cross-model DFlash draft
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
optimization candidate, not a default — acceptance has to be measured on this
traffic first. The bats seam fails if a draft flag ships unannounced.

## Rollback

|Target|How|
|---|---|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|

## When it misbehaves

- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
  naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests queue behind each other: three concurrent long generations are the
  slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
  for the three, 3.0 s for the fourth).
- `400 … exceeds the available context size (180224 tokens)`: the conversation
  outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
  and the per-slot window is the binding limit. Shorten the prompt, compact, or
  roll back to the 262,144 shape above.
- Vision answers a color it never saw: the client sent `images` instead of an
  `image_url` content part (see *Client contract*).
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
  of VRAM into unified memory halves the rate silently.