dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-small Q6_K cutover runbook + measured post-boot numbers

Browse Source 
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used /
  2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on
  Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card),
  512px red-PNG vision verified, and the new probe-size law (>=256 px: a
  10x10 image undersupplies image tokens and the model answers 'video'),
  nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides)
- frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and
  becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured)
- AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index
  (active row + two retired rows); LLM + root index entries
- Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical
  to Q5_K_M - the vision probe scare was probe size, not payload
This commit is contained in:
Frank committed 2026-10-03 11:20:33 -07:00
1 parent ff8c0ee264
commit 50459e4e31
5 files changed
+206 -73
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
+23 -21
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves."
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
tags:
- llm
- frank-small
@@ -14,23 +14,23 @@ tags:
timestamp: 2026-10-03
---
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
> **ACTIVE since 2026-10-02, resliced 2026-10-03.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime, resized twice on
> 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of
> the card free), and on 2026-10-03 Frank ordered the third shape: **two
> conversations of 200,704 tokens each with the KV VALUES unquantized** —
> **RETIRED 2026-10-03** — superseded by
> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
> decode is the measured price of the higher rung on this bandwidth-bound
> card. This file remains the rollback reference and the owner of the
> measured boot-pair table.
> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
> conversations of 200,704 tokens with the KV VALUES unquantized** —
> *"2 streams with 200700 tokens context or more… if we have extra space on
> the 3090, unquantize the kv."* The freed third slot pays for exact V, and
> the boot lands **20,530 MiB used / 4,046 MiB free** — above the 2–3 GiB
> floor, with full-precision KV.
> **The tradeoffs of the reslice:** one fewer simultaneous conversation, a
> per-slot window that grew from 180,224 to 200,704 (still below the native
> 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger
> per-token KV traffic. The 3 × 180,224 shape is one env edit away — see
> *Rollback*.
> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
## Identity
@@ -90,7 +90,7 @@ measured band.
|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
|Vision|projector loads at boot (`loaded multimodal model`); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice|
|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
## What the slot geometry costs
@@ -146,7 +146,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
@@ -165,7 +165,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
and the per-slot window is the binding limit. Shorten the prompt, compact,
or roll back to the 262,144 shape above.
- Vision answers a color it never saw: the client sent `images` instead of an
`image_url` content part (see *Client contract*).
- Vision answers a color it never saw, or claims the image "is a video":
either the client sent `images` instead of an `image_url` content part, or
the probe image is too small to supply real image tokens (use >= 256 px;
see *Client contract*).
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
of VRAM into unified memory halves the rate silently.