dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Browse Source 
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
This commit is contained in:
Frank
2026-10-02 03:46:50 -07:00
parent 7fe881de01
commit 934285c2b2
7 changed files with 245 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
+93 -35
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves."
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
tags:
- llm
- frank-small
@@ -17,11 +17,17 @@ timestamp: 2026-10-02
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
> **ACTIVE since 2026-10-02.** One quant step down from
> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the
> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M`
> on llama.cpp, two contexts, whatever context size fits the 3090. The answer
> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the
> model's native window and two of them already fit.
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
> contract, then **resized twice the same day** on Frank's orders: first to
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
> card free. Frank's arithmetic request — "as much as can fit while keeping
> overhead open space to 2-3gb on the 3090" — is answered by the two
> measurements in *Shape and the KV contract* below; the answer is
> **3 × 180,224**, not 3 × 262,144.
> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
> instead of the model's native 262,144. See *What three slots cost*.
## Identity
@@ -29,15 +35,16 @@ timestamp: 2026-10-02
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
## Shape and the KV contract
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so
each session gets `n_ctx_slot = 262144`, the model's full native window (boot
log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
`PARALLEL`; the bats seam enforces it.
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
@@ -47,29 +54,64 @@ The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
lie. The serve script pins the trio; the bats seam pins the script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈
**12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) +
compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's
24,576 MiB.
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
**Why the quant step did not buy a bigger window.** The model cannot be run
past its trained 262,144 without scaling, and two of those windows are already
the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not
geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token
instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That
arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or
quant change that breaks the fit fails the suite instead of the boot.
**The pool size came from two boots, not from a formula.** `free`-style
arithmetic on this stack is unreliable because the compute buffers, CUDA
graphs and now the vision tower all bill separately, so the fit was measured
at two shapes and solved between them:
## Measured on this file (2026-10-02)
|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|---|---|---|---|---|
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
The two rows price the projector and its image buffers at **~1.4 GiB** — not
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
activation and compute buffers ride along. Solving the second row's non-KV cost
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
**2,441 MiB free** (measured, 21,684 MiB used).
**Why 3 × 262,144 was never possible.** Three native windows are 786,432
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
6 GiB.
## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
|Check|Result|
|---|---|
|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`|
|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)|
|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)|
|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)|
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)|
|Thinking|`reasoning_content` populated on a reasoning-requiring prompt|
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|Prefill|**3,814 tok/s** (same request)|
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
scan at equal prompt depth, and one less slot's DeltaNet state resident.
## What three slots cost
The per-slot window is the whole price. Two consequences the fleet depends on:
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
sets `contextWindow: 180224` for this endpoint — the harness compacts before
it reaches the ceiling instead of discovering it.
- **frank-primary's compaction prompt no longer fits with room to answer.**
Its trigger is 183,501 tokens — already above this slot — and its summary
budget is 32,768 output tokens. On the two-slot shape (262,144) that left
32.8k of output room, which is why that number was chosen; on this shape a
single compaction request must be truncated or split. Nothing in the fleet
routes compaction here automatically, but any runbook that claims frank-small
"fits a compaction request in ONE request" is describing the 262,144 shape.
## Client contract
@@ -81,10 +123,17 @@ quant change that breaks the fit fails the suite instead of the boot.
`extraBody`. A raw client that omits it gets a direct answer — measured, and
not a fault. Trivial questions produce no thinking even with the flag on.
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a
paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one
`--mmproj` away if it is ever wanted here (frank-primary is the fleet's
vision endpoint today).
- **Vision is on**, through the projector file (`--mmproj`). Use the
OpenAI-standard `image_url` content part — that path answers correctly
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
read by `/v1/chat/completions`: the same request sent that way answered
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
but refuses a request without `messages` in this build, so chat is the only
vision path.
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
refusal, never a silent fallback to blindness.
## No spec decode
@@ -98,7 +147,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|Target|How|
|---|---|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
## When it misbehaves
@@ -107,7 +158,14 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests queue behind each other: two concurrent long generations are the
slot ceiling by contract.
- Requests queue behind each other: three concurrent long generations are the
slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
for the three, 3.0 s for the fourth).
- `400 … exceeds the available context size (180224 tokens)`: the conversation
outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
and the per-slot window is the binding limit. Shorten the prompt, compact, or
roll back to the 262,144 shape above.
- Vision answers a color it never saw: the client sent `images` instead of an
`image_url` content part (see *Client contract*).
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
of VRAM into unified memory halves the rate silently.
serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md
+4 -1
View File
@@ -52,7 +52,10 @@ timestamp: 2026-10-01
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
one request. A single request may grow to the whole pool — a
170,019-token prompt was verified at the 4-slot shape — the grid is
the two-way concurrency contract.
the two-way concurrency contract. **That compaction arithmetic is
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
180,224, which 183,501 tokens already exceeds — see *What three slots
cost* in [[frank-small-llama-mimo-q5km]].
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
serverAppConfigs-docs/LLM/index.md
+3 -2
View File
@@ -76,8 +76,9 @@ frank-small, port 7771]
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork