docs: frank-small Q6_K cutover runbook + measured post-boot numbers
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used / 2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card), 512px red-PNG vision verified, and the new probe-size law (>=256 px: a 10x10 image undersupplies image tokens and the model answers 'video'), nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides) - frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured) - AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index (active row + two retired rows); LLM + root index entries - Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical to Q5_K_M - the vision probe scare was probe size, not payload
This commit is contained in:
Frank
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 20,530 used / **4,046 free** of 24,576 MiB, 104.8 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V: the freed slot paid for exact values); rollback `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` or `models/MiMo-Q6K/` | 7771 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 21,276 used / **2,849 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -289,7 +289,7 @@ of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
|
||||
`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
|
||||
`frank-primary` through a BF16 mmproj loaded explicitly (this build never
|
||||
autoloads projectors), and `frank-small` through
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M
|
||||
`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
|
||||
GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
|
||||
**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the
|
||||
model's native 262144: the harness must compact before llama.cpp answers a
|
||||
@@ -389,10 +389,13 @@ Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
|
||||
sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on
|
||||
port 7771. The runtime arrived 2026-10-01 night for one reason:
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B,
|
||||
sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`**
|
||||
(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the
|
||||
projector is a hardlink of the Q5KM copy — the dir is self-contained), on
|
||||
port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the
|
||||
reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for
|
||||
one reason:
|
||||
**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's
|
||||
only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V
|
||||
together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice
|
||||
@@ -411,30 +414,38 @@ drives that term.
|
||||
conversations of 200,704 tokens each** (llama.cpp divides the pool:
|
||||
`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
|
||||
**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch
|
||||
2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **20,530
|
||||
used / 4,046 free** of 24,576 MiB — above the 2–3 GiB floor Frank asked for,
|
||||
solved from the boot-pair table in the runbook (non-KV cost ~7,986 MiB at this
|
||||
shape; f16 K+V is 32 KiB/token across the 8 attention layers). The 2026-10-02
|
||||
3 × 180,224 + q8_0-V shape stays one env edit away (`CTX_SIZE=540672
|
||||
PARALLEL=3 CACHE_TYPE_V=q8_0`). Three native windows need 18.4 GiB of KV alone
|
||||
and were never on the table.
|
||||
2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **21,276
|
||||
used / 2,849 free** of 24,576 MiB — inside the 2–3 GiB floor Frank asked for
|
||||
(non-KV 8,732 MiB; f16 K+V is 32 KiB/token across the 8 attention layers;
|
||||
quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
|
||||
~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
|
||||
q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
|
||||
CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor
|
||||
there). Three native windows need 18.4 GiB of KV alone and were never on the
|
||||
table.
|
||||
|
||||
Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704`
|
||||
and `loaded multimodal model`; decode **104.84 tok/s** at temp 0 — ~2 % under
|
||||
the 3-slot q8_0-V shape (106.8), the honest price of exact V; health 200 in
|
||||
~5 s; native `tool_calls`; thinking in `reasoning_content`; a solid-red PNG
|
||||
answers "Red." through an `image_url` content part (the legacy top-level
|
||||
`images` field is ignored by `/v1/chat/completions` — the model then invents a
|
||||
color). A prompt over the slot is a **400 `exceed_context_size_error`**, which
|
||||
is why `models.yml` sets `contextWindow: 200704`.
|
||||
and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe
|
||||
that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**:
|
||||
decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more
|
||||
bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in
|
||||
`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an
|
||||
`image_url` content part (two traps: the legacy top-level `images` field is
|
||||
ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made
|
||||
the model claim "it's a video"; probe with ≥256 px). A prompt over the slot
|
||||
is a **400 `exceed_context_size_error`**, which is why `models.yml` sets
|
||||
`contextWindow: 200704`.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: the previous shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
|
||||
(3 × 180,224, measured 2,441 MiB free); before that
|
||||
[[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`) and [[frank-small-sglang-mimo-v26]] (venv on
|
||||
disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md`
|
||||
(the boot-pair table that prices pool changes lives in the retired
|
||||
[[frank-small-llama-mimo-q5km]]).
|
||||
Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf`
|
||||
buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both
|
||||
measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0`
|
||||
pairs with Q5 only; SGLang rollback in
|
||||
[[frank-small-sglang-mimo-v26]] (venv on disk at
|
||||
`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE)
|
||||
|
||||
@@ -838,9 +849,11 @@ linked runbooks.
|
||||
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
|
||||
The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
|
||||
pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
|
||||
decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. Both
|
||||
moves measured; both directions cost or buy window, never pool. See
|
||||
[[frank-small-llama-mimo-q5km]].
|
||||
decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
|
||||
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
|
||||
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
|
||||
All moves measured; both slot directions cost or buy window, never pool.
|
||||
See [[frank-small-llama-mimo-q6k-2x200704]].
|
||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||
costs ~1.5x its file size.** The frank-small 3090 fit was solved from two
|
||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||
@@ -960,7 +973,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -972,7 +985,8 @@ Retired (rollback targets and history):
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves."
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||||
description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -14,23 +14,23 @@ tags:
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||||
|
||||
> **ACTIVE since 2026-10-02, resliced 2026-10-03.** One quant step down from
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime, resized twice on
|
||||
> 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of
|
||||
> the card free), and on 2026-10-03 Frank ordered the third shape: **two
|
||||
> conversations of 200,704 tokens each with the KV VALUES unquantized** —
|
||||
> **RETIRED 2026-10-03** — superseded by
|
||||
> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||||
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
|
||||
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
|
||||
> decode is the measured price of the higher rung on this bandwidth-bound
|
||||
> card. This file remains the rollback reference and the owner of the
|
||||
> measured boot-pair table.
|
||||
|
||||
> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]]
|
||||
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
|
||||
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
|
||||
> conversations of 200,704 tokens with the KV VALUES unquantized** —
|
||||
> *"2 streams with 200700 tokens context or more… if we have extra space on
|
||||
> the 3090, unquantize the kv."* The freed third slot pays for exact V, and
|
||||
> the boot lands **20,530 MiB used / 4,046 MiB free** — above the 2–3 GiB
|
||||
> floor, with full-precision KV.
|
||||
|
||||
> **The tradeoffs of the reslice:** one fewer simultaneous conversation, a
|
||||
> per-slot window that grew from 180,224 to 200,704 (still below the native
|
||||
> 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger
|
||||
> per-token KV traffic. The 3 × 180,224 shape is one env edit away — see
|
||||
> *Rollback*.
|
||||
> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
|
||||
> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -90,7 +90,7 @@ measured band.
|
||||
|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
|
||||
|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
|
||||
|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
|
||||
|Vision|projector loads at boot (`loaded multimodal model`); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice|
|
||||
|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
|
||||
|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
|
||||
|
||||
## What the slot geometry costs
|
||||
@@ -146,7 +146,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|
||||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`|
|
||||
@@ -165,7 +165,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
|
||||
and the per-slot window is the binding limit. Shorten the prompt, compact,
|
||||
or roll back to the 262,144 shape above.
|
||||
- Vision answers a color it never saw: the client sent `images` instead of an
|
||||
`image_url` content part (see *Client contract*).
|
||||
- Vision answers a color it never saw, or claims the image "is a video":
|
||||
either the client sent `images` instead of an `image_url` content part, or
|
||||
the probe image is too small to supply real image tokens (use >= 256 px;
|
||||
see *Client contract*).
|
||||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||||
of VRAM into unified memory halves the rate silently.
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV"
|
||||
description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
timestamp: 2026-10-03
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on
|
||||
> this card: the freed third slot paid for **exact KV values**
|
||||
> ([[frank-small-llama-mimo-q5km]] documents that step and remains the
|
||||
> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung**
|
||||
> — the same file this endpoint served on 2026-10-01, come back as an
|
||||
> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704);
|
||||
> only the weights moved up.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
|
||||
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||||
so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
|
||||
divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
|
||||
(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
|
||||
ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).
|
||||
|
||||
**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
|
||||
(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
|
||||
Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
|
||||
landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
|
||||
inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
|
||||
12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
|
||||
field, never `total − used`: the driver holds ~450 MiB of reserve the
|
||||
subtraction hides (that error once overstated this card's slack by a
|
||||
half-GiB).
|
||||
|
||||
**Pool prices on this card come from boots, never formulas** — the measured
|
||||
boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
|
||||
contract*) and still prices every rollback shape.
|
||||
|
||||
## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
|
||||
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung|
|
||||
|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
|
||||
|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
|
||||
|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
|
||||
|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface, live-verified on this model family: model id `frank-small`,
|
||||
native tools via `--jinja` (`tool_calls` in the response), thinking in
|
||||
**`reasoning_content`**, vision via `image_url` content parts. Two vision
|
||||
traps, both measured here:
|
||||
|
||||
1. A top-level `images` field on `/v1/chat/completions` is silently ignored
|
||||
and the model invents a color.
|
||||
2. **The probe image must be big enough to supply real image tokens.** A
|
||||
10×10 px solid-red PNG came back "The image is a video, so I can't see its
|
||||
content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
|
||||
tokens, and a postage-stamp image lands one patch below usable signal.
|
||||
512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
|
||||
image; a wrong-color answer from a tiny image is the probe's fault, not
|
||||
the endpoint's.
|
||||
|
||||
`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
|
||||
so the harness compacts before llama.cpp answers a longer prompt with a 400.
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|
||||
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*):
|
||||
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
|
||||
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
|
||||
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
|
||||
behind two long generations ⇒ the slot ceiling is two, by contract.
|
||||
@@ -19,7 +19,7 @@ kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
|
||||
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can
|
||||
read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE
|
||||
session at 262,144, f16 KV, vision via the BF16 projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
@@ -81,13 +81,18 @@ frank-small, port 7771]
|
||||
ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the
|
||||
hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on
|
||||
disk
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced
|
||||
2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K
|
||||
+ f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the
|
||||
card left free. The boot-pair pool arithmetic, the slot-window contract,
|
||||
rollback to 3 x 180,224 or Q6_K
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
|
||||
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
|
||||
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
|
||||
measured -10 % vs Q5_K_M: weight-bandwidth-bound), 2,849 MiB of the card
|
||||
left free. Probe-size trap (>=256 px images or the model says "video")
|
||||
documented. Rollback: one MODEL_FILE line back to Q5_K_M
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
@@ -252,8 +257,10 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target)**: llama.cpp (master, CUDA sm_86 build) serving
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||||
[[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
|
||||
@@ -288,16 +288,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
|
||||
2026-10-02, resliced 2026-10-03: a CUDA (sm_86) build of llama.cpp master
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's
|
||||
RTX 3090, port 7771 — TWO conversations at 200,704 tokens each with fully
|
||||
unquantized f16 KV, vision through the repo's mmproj, and the boot-pair
|
||||
arithmetic that sizes the pool; 104.8 tok/s decode, 4,046 MiB free
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
|
||||
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
|
||||
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
|
||||
2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
|
||||
since 2026-10-03: llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
|
||||
port 7771 — TWO conversations at 200,704 tokens each with fully
|
||||
unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
|
||||
costs a measured −10 % vs Q5_K_M — weight-bandwidth), 2,849 MiB free, and
|
||||
the ≥256 px image-probe trap (tiny images answer "video")
|
||||
- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target, payload on disk; owns the measured boot-pair table): same runtime
|
||||
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
|
||||
Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
|
||||
[[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
|
||||
for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
|
||||
SGLang in a self-contained python3.12 venv serving
|
||||
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
|
||||
|
||||
Reference in New Issue
Block a user