docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
This commit is contained in:
Frank
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) on the RTX 3090, **2 × 262,144** (native window) with **asymmetric KV f16 K + q8_0 V** under flash attention, 20,164/24,576 MiB VRAM, 100.4–101.8 tok/s decode, text-only. Swapped 2026-10-02 from Q6_K; rollback `models/MiMo-Q6K/` | 7771 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP.
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, SGLang on the 3090):
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
@@ -277,12 +277,15 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-02 `frank-large` (the BF16 mmproj that ships inside its GGUF
|
||||
payload) and `frank-primary` (loads the BF16 mmproj explicitly — this build
|
||||
never autoloads projectors) accept images (`input: [text, image]`);
|
||||
`frank-small` is **text-only** — the GGUF payload it moved to on 2026-10-02
|
||||
ships no projector, and the in-checkpoint MiMo vision tower exists only in the
|
||||
safetensors layouts. Note also that
|
||||
of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
|
||||
`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
|
||||
`frank-primary` through a BF16 mmproj loaded explicitly (this build never
|
||||
autoloads projectors), and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
**180224**, the per-slot window, not the model's native 262144: the harness
|
||||
must compact before llama.cpp answers a longer prompt with a 400. Note also
|
||||
that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
models.yml sets `reasoningContentField` per endpoint to match.
|
||||
@@ -370,11 +373,14 @@ unit/env/serve trio and redeploy.
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
|
||||
sha256 `04cf46f2…`) on port 7771. The 2026-10-02 swap was a **quant rung down
|
||||
from Q6_K**, not a runtime change — the llama.cpp path arrived 2026-10-01
|
||||
night for one reason: **asymmetric KV**. `--cache-type-k f16` +
|
||||
`--cache-type-v q8_0` is the fleet's only per-tensor KV lever; SGLang's
|
||||
`--kv-cache-dtype` moves K and V together.
|
||||
sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
|
||||
port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a
|
||||
runtime change — the llama.cpp path arrived 2026-10-01 night for one reason:
|
||||
**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together.
|
||||
Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so
|
||||
the projector file is the entire vision path (and ~1.4 GiB of the card).
|
||||
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
@@ -382,19 +388,30 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
|
||||
**The shape:** `--ctx-size 524288 --parallel 2` = TWO full native windows
|
||||
(llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`),
|
||||
`--flash-attn on` (the q8_0-V path is only cheap under flash attention — the
|
||||
seam pins it), f16 K + q8_0 V, all layers on the GPU. Measured VRAM
|
||||
**20,164 / 24,576 MiB**; full f16 V would add ~3.96 GiB and bust the card,
|
||||
which is the whole justification for the asymmetric contract.
|
||||
**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of
|
||||
180,224 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is
|
||||
only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all
|
||||
layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441
|
||||
free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by
|
||||
solving the pool against two measured boots: non-KV cost is 7,390 MiB without
|
||||
a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so
|
||||
vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native
|
||||
windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the
|
||||
table. Full f16 V would add ~4 GiB more and bust the card — that is the
|
||||
justification for the asymmetric contract.
|
||||
|
||||
Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; decode
|
||||
**100.4–101.8 tok/s** at temp 0; two concurrent requests answered in
|
||||
parallel; native `tool_calls`; thinking in `reasoning_content`. **Text-only**
|
||||
— this GGUF ships no projector, so `models.yml` declares `input: [text]`
|
||||
(the in-checkpoint vision tower exists only in the safetensors layouts the
|
||||
SGLang run used).
|
||||
Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs
|
||||
`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode
|
||||
**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the
|
||||
2 × 262,144 shape, because a shorter slot means a shorter attention scan;
|
||||
three concurrent requests answered in parallel and a fourth queued behind a
|
||||
released slot; native `tool_calls`; thinking in `reasoning_content`; a
|
||||
solid-red PNG answers "Red." through an `image_url` content part (the legacy
|
||||
top-level `images` field is ignored by `/v1/chat/completions` — the model then
|
||||
invents a color). A prompt over the slot is a **400
|
||||
`exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 180224`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
@@ -701,6 +718,35 @@ linked runbooks.
|
||||
host-RAM hash table) and its window decides which traffic it can see: too
|
||||
short and it proposes on ordinary prose fragments, which costs verification
|
||||
without acceptance.
|
||||
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
|
||||
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
|
||||
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
|
||||
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
|
||||
available context size (180224 tokens)`) even when the pool has room. So
|
||||
frank-small's 2 → 3 reslice moved each conversation from the model's native
|
||||
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
|
||||
the harness would discover the ceiling as a 400 on a long conversation. Side
|
||||
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
||||
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
|
||||
Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]].
|
||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||
costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
|
||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
|
||||
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
|
||||
that bills ~1.4 GiB, because the encoder's activation and compute buffers
|
||||
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
|
||||
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
|
||||
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
||||
`PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize
|
||||
starts from measurement.
|
||||
- **`images` and `image_url` are NOT interchangeable: only the content part
|
||||
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
||||
model still invents an answer if the client used the legacy top-level
|
||||
`images` field on `/v1/chat/completions` — frank-small answered "Brown." for
|
||||
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
||||
failure is silent because a hallucinated color is a plausible answer. Probe
|
||||
vision with a solid color and expect an exact word.
|
||||
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
||||
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
||||
then served frank-large: every tool-using agent on that endpoint was broken
|
||||
|
||||
@@ -13,8 +13,9 @@
|
||||
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
||||
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
|
||||
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
|
||||
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB used,
|
||||
# ~2.7 GiB free, the middle of what was asked for.
|
||||
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
|
||||
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
|
||||
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
|
||||
#
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves."
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -17,11 +17,17 @@ timestamp: 2026-10-02
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** One quant step down from
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the
|
||||
> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M`
|
||||
> on llama.cpp, two contexts, whatever context size fits the 3090. The answer
|
||||
> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the
|
||||
> model's native window and two of them already fit.
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV
|
||||
> contract, then **resized twice the same day** on Frank's orders: first to
|
||||
> three contexts with vision, then down to the pool that keeps 2–3 GiB of the
|
||||
> card free. Frank's arithmetic request — "as much as can fit while keeping
|
||||
> overhead open space to 2-3gb on the 3090" — is answered by the two
|
||||
> measurements in *Shape and the KV contract* below; the answer is
|
||||
> **3 × 180,224**, not 3 × 262,144.
|
||||
|
||||
> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens
|
||||
> of per-conversation memory versus the two-slot shape: 180,224 tokens each
|
||||
> instead of the model's native 262,144. See *What three slots cost*.
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -29,15 +35,16 @@ timestamp: 2026-10-02
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so
|
||||
each session gets `n_ctx_slot = 262144`, the model's full native window (boot
|
||||
log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||||
`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so
|
||||
each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3,
|
||||
n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by
|
||||
`PARALLEL`; the bats seam enforces it.
|
||||
|
||||
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
@@ -47,29 +54,64 @@ The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||||
lie. The serve script pins the trio; the bats seam pins the script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈
|
||||
**12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) +
|
||||
compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's
|
||||
24,576 MiB.
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool.
|
||||
|
||||
**Why the quant step did not buy a bigger window.** The model cannot be run
|
||||
past its trained 262,144 without scaling, and two of those windows are already
|
||||
the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not
|
||||
geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token
|
||||
instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That
|
||||
arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or
|
||||
quant change that breaks the fit fails the suite instead of the boot.
|
||||
**The pool size came from two boots, not from a formula.** `free`-style
|
||||
arithmetic on this stack is unreliable because the compute buffers, CUDA
|
||||
graphs and now the vision tower all bill separately, so the fit was measured
|
||||
at two shapes and solved between them:
|
||||
|
||||
## Measured on this file (2026-10-02)
|
||||
|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)|
|
||||
|---|---|---|---|---|
|
||||
|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**|
|
||||
|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**|
|
||||
|
||||
The two rows price the projector and its image buffers at **~1.4 GiB** — not
|
||||
the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own
|
||||
activation and compute buffers ride along. Solving the second row's non-KV cost
|
||||
against Frank's 2–3 GiB band: the card is 24,576 MiB, so
|
||||
`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded
|
||||
down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and
|
||||
**2,441 MiB free** (measured, 21,684 MiB used).
|
||||
|
||||
**Why 3 × 262,144 was never possible.** Three native windows are 786,432
|
||||
tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is
|
||||
27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed
|
||||
6 GiB.
|
||||
|
||||
## Measured on this shape (2026-10-02, 3 × 180,224 + vision)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`|
|
||||
|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)|
|
||||
|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)|
|
||||
|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)|
|
||||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)|
|
||||
|Thinking|`reasoning_content` populated on a reasoning-requiring prompt|
|
||||
|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged|
|
||||
|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)|
|
||||
|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)|
|
||||
|Prefill|**3,814 tok/s** (same request)|
|
||||
|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots|
|
||||
|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires|
|
||||
|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**|
|
||||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)|
|
||||
|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted|
|
||||
|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated|
|
||||
|
||||
Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same
|
||||
day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention
|
||||
scan at equal prompt depth, and one less slot's DeltaNet state resident.
|
||||
|
||||
## What three slots cost
|
||||
|
||||
The per-slot window is the whole price. Two consequences the fleet depends on:
|
||||
|
||||
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||||
sets `contextWindow: 180224` for this endpoint — the harness compacts before
|
||||
it reaches the ceiling instead of discovering it.
|
||||
- **frank-primary's compaction prompt no longer fits with room to answer.**
|
||||
Its trigger is 183,501 tokens — already above this slot — and its summary
|
||||
budget is 32,768 output tokens. On the two-slot shape (262,144) that left
|
||||
32.8k of output room, which is why that number was chosen; on this shape a
|
||||
single compaction request must be truncated or split. Nothing in the fleet
|
||||
routes compaction here automatically, but any runbook that claims frank-small
|
||||
"fits a compaction request in ONE request" is describing the 262,144 shape.
|
||||
|
||||
## Client contract
|
||||
|
||||
@@ -81,10 +123,17 @@ quant change that breaks the fit fails the suite instead of the boot.
|
||||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||||
- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a
|
||||
paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one
|
||||
`--mmproj` away if it is ever wanted here (frank-primary is the fleet's
|
||||
vision endpoint today).
|
||||
- **Vision is on**, through the projector file (`--mmproj`). Use the
|
||||
OpenAI-standard `image_url` content part — that path answers correctly
|
||||
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
|
||||
read by `/v1/chat/completions`: the same request sent that way answered
|
||||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||||
but refuses a request without `messages` in this build, so chat is the only
|
||||
vision path.
|
||||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops
|
||||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||||
refusal, never a silent fallback to blindness.
|
||||
|
||||
## No spec decode
|
||||
|
||||
@@ -98,7 +147,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|
||||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
@@ -107,7 +158,14 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests queue behind each other: two concurrent long generations are the
|
||||
slot ceiling by contract.
|
||||
- Requests queue behind each other: three concurrent long generations are the
|
||||
slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s
|
||||
for the three, 3.0 s for the fourth).
|
||||
- `400 … exceeds the available context size (180224 tokens)`: the conversation
|
||||
outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens
|
||||
and the per-slot window is the binding limit. Shorten the prompt, compact, or
|
||||
roll back to the 262,144 shape above.
|
||||
- Vision answers a color it never saw: the client sent `images` instead of an
|
||||
`image_url` content part (see *Client contract*).
|
||||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||||
of VRAM into unified memory halves the rate silently.
|
||||
|
||||
@@ -52,7 +52,10 @@ timestamp: 2026-10-01
|
||||
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
|
||||
one request. A single request may grow to the whole pool — a
|
||||
170,019-token prompt was verified at the 4-slot shape — the grid is
|
||||
the two-way concurrency contract.
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
|
||||
180,224, which 183,501 tokens already exceeds — see *What three slots
|
||||
cost* in [[frank-small-llama-mimo-q5km]].
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
|
||||
@@ -76,8 +76,9 @@ frank-small, port 7771]
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
|
||||
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
@@ -285,8 +285,9 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
|
||||
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
|
||||
3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract
|
||||
(f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode
|
||||
3090, port 7771 — THREE conversations at 180,224 tokens each, the
|
||||
asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
|
||||
and the two VRAM measurements that set the pool size; ~100 tok/s decode
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
|
||||
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
|
||||
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
|
||||
|
||||
@@ -925,3 +925,71 @@ any repo service.
|
||||
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
|
||||
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
|
||||
drafter, not a remnant).
|
||||
|
||||
## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
|
||||
|
||||
Frank's order was the highest quality of
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
|
||||
runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
|
||||
carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
|
||||
llama.cpp, gufo and every other loader on the fleet cannot read — only the
|
||||
kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
|
||||
`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
|
||||
bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||||
(n-max 4, `--spec-draft-device ROCm0`).
|
||||
|
||||
- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
|
||||
(+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
|
||||
mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
|
||||
routing variant was chosen over the card's plain Q8_0 because it keeps 187
|
||||
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
|
||||
0.953 vs 0.911.
|
||||
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
|
||||
- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
|
||||
fetchers moved to `old/`. Runbook
|
||||
[[frank-large-rocmfpx-q8agent-mtp]].
|
||||
|
||||
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
|
||||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||||
with two contexts ("whatever context size fits the 3090"), then **three
|
||||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||||
the 3090", then **"frank-small should have vision"**.
|
||||
|
||||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||||
**21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
|
||||
- **The pool came from two boots, not a formula.** 2 x 262,144 with no
|
||||
projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
|
||||
with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
|
||||
difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
|
||||
encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
|
||||
booted healthy but left only 1,153 MiB free, which is how the band got
|
||||
violated and why the seam now prices non-KV at the measured 9 GiB and
|
||||
REJECTS that shape.
|
||||
- **Three contexts cost window, and that is the tradeoff Frank was told:**
|
||||
180,224 tokens per conversation, not the native 262,144. A longer prompt is
|
||||
a hard `400 exceed_context_size_error`, so `models.yml` sets
|
||||
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
|
||||
compaction request no longer fits here with summary room; the two-slot
|
||||
rollback is the way back.
|
||||
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
|
||||
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
|
||||
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
|
||||
queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
|
||||
`reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
|
||||
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
|
||||
top-level `images` field — that request answered "Brown." for the same red
|
||||
image, i.e. it generated from text alone. Only the `image_url` content part
|
||||
reaches the model. `/completion` accepts `images` but this build still
|
||||
requires `messages`.
|
||||
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
|
||||
mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
|
||||
empty) in the env file is the documented text-only opt-out and returns ~1.4
|
||||
GiB to KV.
|
||||
- Reclaim staged, **not purged, awaiting approval**:
|
||||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
|
||||
Reference in New Issue
Block a user