docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115 tensor-type lock-in that forced the fork, the pinned build (set_rows history), the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live client contract (reasoning_content, effort fold, vision restored) and rollback. frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired (rollback target). gufo runbook marked retired (rollback target). Both indexes, the fleet topology table, the vision/models.yml notes, the benchmarking law and three new Key Lessons updated.
This commit is contained in:
Frank committed
1 parent
125d04967f
commit
72bd4c967e
7 files changed
+522
-147
No files matched your search
@@ -0,0 +1,113 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
- mimo
|
||||
- gguf
|
||||
- kv-cache
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** One quant step down from
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the
|
||||
> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M`
|
||||
> on llama.cpp, two contexts, whatever context size fits the 3090. The answer
|
||||
> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the
|
||||
> model's native window and two of them already fit.
|
||||
|
||||
## Identity
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so
|
||||
each session gets `n_ctx_slot = 262144`, the model's full native window (boot
|
||||
log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||||
|
||||
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
|
||||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
|
||||
Without the flag the engine falls back silently and the contract becomes a
|
||||
lie. The serve script pins the trio; the bats seam pins the script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈
|
||||
**12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) +
|
||||
compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's
|
||||
24,576 MiB.
|
||||
|
||||
**Why the quant step did not buy a bigger window.** The model cannot be run
|
||||
past its trained 262,144 without scaling, and two of those windows are already
|
||||
the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not
|
||||
geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token
|
||||
instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That
|
||||
arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or
|
||||
quant change that breaks the fit fails the suite instead of the boot.
|
||||
|
||||
## Measured on this file (2026-10-02)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`|
|
||||
|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)|
|
||||
|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)|
|
||||
|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)|
|
||||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)|
|
||||
|Thinking|`reasoning_content` populated on a reasoning-requiring prompt|
|
||||
|
||||
## Client contract
|
||||
|
||||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
|
||||
the field `models.yml` reads for this endpoint.
|
||||
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
|
||||
`enable_thinking` false, so `models.yml` ships
|
||||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
|
||||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||||
- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a
|
||||
paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one
|
||||
`--mmproj` away if it is ever wanted here (frank-primary is the fleet's
|
||||
vision endpoint today).
|
||||
|
||||
## No spec decode
|
||||
|
||||
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
|
||||
ghost), and the cross-model DFlash draft
|
||||
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
|
||||
optimization candidate, not a default — acceptance has to be measured on this
|
||||
traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests queue behind each other: two concurrent long generations are the
|
||||
slot ceiling by contract.
|
||||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||||
of VRAM into unified memory halves the rate silently.
|
||||
Reference in new issue
Block a user