---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves."
tags:
  - llm
  - frank-small
  - dankclaw
  - llama-cpp
  - rtx3090
  - mimo
  - gguf
  - kv-cache
timestamp: 2026-10-02
---

# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)

> **ACTIVE since 2026-10-02.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the
> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M`
> on llama.cpp, two contexts, whatever context size fits the 3090. The answer
> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the
> model's native window and two of them already fit.

## Identity

|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)|
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|

## Shape and the KV contract

`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so
each session gets `n_ctx_slot = 262144`, the model's full native window (boot
log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).

The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:

- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
  Without the flag the engine falls back silently and the contract becomes a
  lie. The serve script pins the trio; the bats seam pins the script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
  (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈
  **12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) +
  compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's
  24,576 MiB.

**Why the quant step did not buy a bigger window.** The model cannot be run
past its trained 262,144 without scaling, and two of those windows are already
the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not
geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token
instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That
arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or
quant change that breaks the fit fails the suite instead of the boot.

## Measured on this file (2026-10-02)

|Check|Result|
|---|---|
|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`|
|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)|
|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)|
|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)|
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)|
|Thinking|`reasoning_content` populated on a reasoning-requiring prompt|

## Client contract

- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
  the field `models.yml` reads for this endpoint.
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
  `enable_thinking` false, so `models.yml` ships
  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
  `extraBody`. A raw client that omits it gets a direct answer — measured, and
  not a fault. Trivial questions produce no thinking even with the flag on.
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a
  paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one
  `--mmproj` away if it is ever wanted here (frank-primary is the fleet's
  vision endpoint today).

## No spec decode

The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
ghost), and the cross-model DFlash draft
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
optimization candidate, not a default — acceptance has to be measured on this
traffic first. The bats seam fails if a draft flag ships unannounced.

## Rollback

|Target|How|
|---|---|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`|

## When it misbehaves

- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
  naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests queue behind each other: two concurrent long generations are the
  slot ceiling by contract.
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
  of VRAM into unified memory halves the rate silently.