docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it

AGENTS.md

@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
-| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) on the RTX 3090, **2 × 262,144** (native window) with **asymmetric KV f16 K + q8_0 V** under flash attention, 20,164/24,576 MiB VRAM, 100.4–101.8 tok/s decode, text-only. Swapped 2026-10-02 from Q6_K; rollback `models/MiMo-Q6K/` | 7771 |
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP.
 1. Stop and mask (self-healing would restart it):
    - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, SGLang on the 3090):
+   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
    - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
      `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
@@ -277,12 +277,15 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-02 `frank-large` (the BF16 mmproj that ships inside its GGUF
-payload) and `frank-primary` (loads the BF16 mmproj explicitly — this build
-never autoloads projectors) accept images (`input: [text, image]`);
-`frank-small` is **text-only** — the GGUF payload it moved to on 2026-10-02
-ships no projector, and the in-checkpoint MiMo vision tower exists only in the
-safetensors layouts. Note also that
+of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
+`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
+`frank-primary` through a BF16 mmproj loaded explicitly (this build never
+autoloads projectors), and `frank-small` through
+`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
+GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
+**180224**, the per-slot window, not the model's native 262144: the harness
+must compact before llama.cpp answers a longer prompt with a 400. Note also
+that
 the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
 models.yml sets `reasoningContentField` per endpoint to match.
@@ -370,11 +373,14 @@ unit/env/serve trio and redeploy.
 **root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
 `/etc/systemd/system/`), serving
 `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
-sha256 `04cf46f2…`) on port 7771. The 2026-10-02 swap was a **quant rung down
-from Q6_K**, not a runtime change — the llama.cpp path arrived 2026-10-01
-night for one reason: **asymmetric KV**. `--cache-type-k f16` +
-`--cache-type-v q8_0` is the fleet's only per-tensor KV lever; SGLang's
-`--kv-cache-dtype` moves K and V together.
+sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
+(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
+port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
+runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
+**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
+only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
+Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
+the projector file is the entire vision path (and ~1.4 GiB of the card).
 
 **Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
 DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
@@ -382,19 +388,30 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
 the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
 drives that term.
 
-**The shape:** `--ctx-size 524288 --parallel 2` = TWO full native windows
-(llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`),
-`--flash-attn on` (the q8_0-V path is only cheap under flash attention — the
-seam pins it), f16 K + q8_0 V, all layers on the GPU. Measured VRAM
-**20,164 / 24,576 MiB**; full f16 V would add ~3.96 GiB and bust the card,
-which is the whole justification for the asymmetric contract.
+**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
+180,224 tokens each** (llama.cpp divides the pool:
+`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
+only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
+layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
+free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
+solving the pool against two measured boots: non-KV cost is 7,390 MiB without
+a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
+vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
+windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
+table. Full f16 V would add ~4 GiB more and bust the card — that is the
+justification for the asymmetric contract.
 
-Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; decode
-**100.4–101.8 tok/s** at temp 0; two concurrent requests answered in
-parallel; native `tool_calls`; thinking in `reasoning_content`. **Text-only**
-— this GGUF ships no projector, so `models.yml` declares `input: [text]`
-(the in-checkpoint vision tower exists only in the safetensors layouts the
-SGLang run used).
+Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
+`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
+**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
+2 × 262,144 shape, because a shorter slot means a shorter attention scan;
+three concurrent requests answered in parallel and a fourth queued behind a
+released slot; native `tool_calls`; thinking in `reasoning_content`; a
+solid-red PNG answers "Red." through an `image_url` content part (the legacy
+top-level `images` field is ignored by `/v1/chat/completions` — the model then
+invents a color). A prompt over the slot is a **400
+`exceed_context_size_error`**, which is why `models.yml` sets
+`contextWindow: 180224`.
 
 Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
 Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
@@ -701,6 +718,35 @@ linked runbooks.
   host-RAM hash table) and its window decides which traffic it can see: too
   short and it proposes on ordinary prose fragments, which costs verification
   without acceptance.
+- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
+  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
+  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
+  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
+  available context size (180224 tokens)`) even when the pool has room. So
+  frank-small's 2 → 3 reslice moved each conversation from the model's native
+  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
+  the harness would discover the ceiling as a 400 on a long conversation. Side
+  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
+  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
+  Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
+- **Price a VRAM budget from two boots, never from a formula — and an mmproj
+  costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
+  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
+  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
+  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
+  that bills ~1.4 GiB, because the encoder's activation and compute buffers
+  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
+  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
+  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
+  `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
+  starts from measurement.
+- **`images` and `image_url` are NOT interchangeable: only the content part
+  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
+  model still invents an answer if the client used the legacy top-level
+  `images` field on `/v1/chat/completions` — frank-small answered "Brown." for
+  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
+  failure is silent because a hallucinated color is a plausible answer. Probe
+  vision with a solid color and expect an exact word.
 - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
   `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
   then served frank-large: every tool-using agent on that endpoint was broken