Frank
656012a0fa
index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and frank-small at 2 x 262144; the active-deployment list had no frank-primary entry at all. Fixed both, plus a malformed GLM list item. log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever measurement, the 2x2 probe table, the measured 1.99x), and the frank-small entry's stale consequence bullet now points at the resolution. frank-small runbook: the compaction-fit note is corrected - the 200704 window moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
172 lines
10 KiB
Markdown
172 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
|
||
tags:
|
||
- llm
|
||
- frank-small
|
||
- dankclaw
|
||
- llama-cpp
|
||
- rtx3090
|
||
- mimo
|
||
- gguf
|
||
- kv-cache
|
||
timestamp: 2026-10-02
|
||
---
|
||
|
||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||
|
||
> **ACTIVE since 2026-10-02.** One quant step down from
|
||
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
|
||
> contract, then **resized twice the same day** on Frank's orders: first to
|
||
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
|
||
> card free. Frank's arithmetic request — "as much as can fit while keeping
|
||
> overhead open space to 2-3gb on the 3090" — is answered by the two
|
||
> measurements in *Shape and the KV contract* below; the answer is
|
||
> **3 × 180,224**, not 3 × 262,144.
|
||
|
||
> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
|
||
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
|
||
> instead of the model's native 262,144. See *What three slots cost*.
|
||
|
||
## Identity
|
||
|
||
|Field|Value|
|
||
|---|---|
|
||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|
||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||
|
||
## Shape and the KV contract
|
||
|
||
`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
|
||
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
|
||
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
|
||
`PARALLEL`; the bats seam enforces it.
|
||
|
||
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||
|
||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
|
||
Without the flag the engine falls back silently and the contract becomes a
|
||
lie. The serve script pins the trio; the bats seam pins the script.
|
||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
|
||
|
||
**The pool size came from two boots, not from a formula.** `free`-style
|
||
arithmetic on this stack is unreliable because the compute buffers, CUDA
|
||
graphs and now the vision tower all bill separately, so the fit was measured
|
||
at two shapes and solved between them:
|
||
|
||
|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|
||
|---|---|---|---|---|
|
||
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|
||
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
|
||
|
||
The two rows price the projector and its image buffers at **~1.4 GiB** — not
|
||
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
|
||
activation and compute buffers ride along. Solving the second row's non-KV cost
|
||
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
|
||
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
|
||
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
|
||
**2,441 MiB free** (measured, 21,684 MiB used).
|
||
|
||
**Why 3 × 262,144 was never possible.** Three native windows are 786,432
|
||
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
|
||
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
|
||
6 GiB.
|
||
|
||
## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
|
||
|
||
|Check|Result|
|
||
|---|---|
|
||
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|
||
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|
||
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|
||
|Prefill|**3,814 tok/s** (same request)|
|
||
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|
||
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|
||
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|
||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|
||
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|
||
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
|
||
|
||
Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
|
||
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
|
||
scan at equal prompt depth, and one less slot's DeltaNet state resident.
|
||
|
||
## What three slots cost
|
||
|
||
The per-slot window is the whole price. Two consequences the fleet depends on:
|
||
|
||
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||
sets `contextWindow: 180224` for this endpoint — the harness compacts before
|
||
it reaches the ceiling instead of discovering it.
|
||
- **frank-primary's compaction prompt fits inside one slot again.** At
|
||
frank-primary's shape-D window (200,704) its compaction trigger is 140,492
|
||
tokens (70 %), so one compaction request lands here with 39,732 tokens of
|
||
room. At the previous 262,144 window the trigger was 183,501 — *above* this
|
||
slot — and the request had to be truncated or split. Nothing in the fleet
|
||
routes compaction here automatically; the point is that the 2026-10-02 window
|
||
trim restored the fit.
|
||
|
||
## Client contract
|
||
|
||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
|
||
the field `models.yml` reads for this endpoint.
|
||
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
|
||
`enable_thinking` false, so `models.yml` ships
|
||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
|
||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||
- **Vision is on**, through the projector file (`--mmproj`). Use the
|
||
OpenAI-standard `image_url` content part — that path answers correctly
|
||
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
|
||
read by `/v1/chat/completions`: the same request sent that way answered
|
||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||
but refuses a request without `messages` in this build, so chat is the only
|
||
vision path.
|
||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
|
||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||
refusal, never a silent fallback to blindness.
|
||
|
||
## No spec decode
|
||
|
||
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
|
||
ghost), and the cross-model DFlash draft
|
||
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
|
||
optimization candidate, not a default — acceptance has to be measured on this
|
||
traffic first. The bats seam fails if a draft flag ships unannounced.
|
||
|
||
## Rollback
|
||
|
||
|Target|How|
|
||
|---|---|
|
||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|
||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|
||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||
|
||
## When it misbehaves
|
||
|
||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
|
||
naming the missing file if the binary or GGUF moved.
|
||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||
- Requests queue behind each other: three concurrent long generations are the
|
||
slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
|
||
for the three, 3.0 s for the fourth).
|
||
- `400 … exceeds the available context size (180224 tokens)`: the conversation
|
||
outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
|
||
and the per-slot window is the binding limit. Shorten the prompt, compact, or
|
||
roll back to the 262,144 shape above.
|
||
- Vision answers a color it never saw: the client sent `images` instead of an
|
||
`image_url` content part (see *Client contract*).
|
||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||
of VRAM into unified memory halves the rate silently.
|