---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the flash-attention dependency that comes with quantized V, the VRAM fit, and what to check when it misbehaves."
tags:
  - llm
  - frank-small
  - dankclaw
  - llama-cpp
  - rtx3090
  - mimo
  - gguf
  - kv-cache
timestamp: 2026-10-01
---

# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)

> **ACTIVE since 2026-10-01 night.** Replaces
> [[frank-small-sglang-mimo-v26]] (SGLang + INT8-W8A16-AutoRound safetensors,
> same architecture). Rolled in by Frank's night order: move this endpoint to
> the GGUF path for **asymmetric KV** — unquantized K + q8_0 V — which only
> llama.cpp exposes per-tensor.

## Identity

|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|

## Shape and the KV contract

`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
so each session gets `n_ctx_slot = 262144`, the model's full native window
(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).

Frank's KV contract: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:

- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
  path. Without the flag the engine falls back silently and the contract
  becomes a lie. The serve script pins the trio; the bats seam pins the
  script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
  (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
  ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
  **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
  order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
  K to fix it.

## Client contract

- Thinking arrives in **`reasoning_content`** (`--reasoning-format
  deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
  `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
- **Thinking is off unless the request asks**: the MiMo GGUF template
  defaults `enable_thinking` false; `models.yml` now ships
  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
  in `extraBody`. A raw client that omits it gets a no-thinking answer
  (measured — this is not a fault).
- Tools: `--jinja` + server-side tool parsing, verified
  `tool_calls` round-trip 2026-10-01 23:0x.
- **Text-only now.** The GGUF carries no vision projector; the
  in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
  `input: [text]`.

## No spec decode (yet)

The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
Q8_0) is an optimization-loop candidate, not a default — `--spec-type
draft-dflash` needs acceptance measured on this traffic first. The bats
seam fails if a draft flag ships unannounced.

## Build

No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
compiler (nvcc 12.4 rejects gcc ≥ 14):

```
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
cmake --build build-cuda -j 16 --target llama-server
```

The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
stay on disk as the rollback: repoint the unit's serve script and env from
git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
use the parent of the 2026-10-01 night swap commit).

## When it misbehaves

- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
  exits 1 naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
  node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests hang at first token while `/health` is 200: both slots busy —
  two concurrent long generations are the ceiling by contract.
- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
  VRAM into unified memory halves the rate silently.