Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
174 lines
11 KiB
Markdown
174 lines
11 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
|
||
description: "The retired 2026-10-02/03 frank-zealot runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- rtx3090
|
||
- mimo
|
||
- gguf
|
||
- kv-cache
|
||
timestamp: 2026-10-03
|
||
---
|
||
|
||
# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
|
||
|
||
> **RETIRED 2026-10-03** — superseded by
|
||
> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
|
||
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
|
||
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
|
||
> decode is the measured price of the higher rung on this bandwidth-bound
|
||
> card. This file remains the rollback reference and the owner of the
|
||
> measured boot-pair table.
|
||
|
||
> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
|
||
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
|
||
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
|
||
> conversations of 200,704 tokens with the KV VALUES unquantized** —
|
||
> *"2 streams with 200700 tokens context or more… if we have extra space on
|
||
> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
|
||
> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
|
||
|
||
## Identity
|
||
|
||
|Field|Value|
|
||
|---|---|
|
||
|Endpoint|`192.168.0.143:7771`, served alias `frank-zealot`|
|
||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
|
||
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|
||
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
|
||
|
||
## Shape and the KV contract
|
||
|
||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||
so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
|
||
`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
|
||
stay divisible by `PARALLEL`; the bats seam enforces it.
|
||
|
||
The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
|
||
(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
|
||
replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
|
||
slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
|
||
V traffic is the measured ~2 % decode tax. The serve script still passes both
|
||
flags explicitly — the asymmetric-era law that a quantized V requires
|
||
`--flash-attn on` stays pinned by the seam.
|
||
|
||
**Pool sizes on this stack come from boots, not formulas.** The compute
|
||
buffers, CUDA graphs and vision tower all bill separately, so every shape
|
||
change is priced from measured rows:
|
||
|
||
|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
|
||
|---|---|---|---|---|
|
||
|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
|
||
|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
|
||
|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
|
||
|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
|
||
|
||
The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
|
||
2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
|
||
of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
|
||
(measured steady-state — the KV pool is allocated whole at boot, so this is
|
||
the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
|
||
an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
|
||
safety margin for window the harness compacts before reaching anyway.
|
||
|
||
**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
|
||
524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
|
||
floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
|
||
measured band.
|
||
|
||
## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
|
||
|
||
|Check|Result|
|
||
|---|---|
|
||
|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
|
||
|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
|
||
|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
|
||
|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
|
||
|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
|
||
|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
|
||
|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
|
||
|
||
## What the slot geometry costs
|
||
|
||
The per-slot window is the whole price, in both directions:
|
||
|
||
- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||
sets `contextWindow: 200704` for this endpoint — the harness compacts before
|
||
it reaches the ceiling instead of discovering it.
|
||
- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
|
||
for frank-dragoon sessions runs on this endpoint (`compactionModel` in
|
||
`models.yml`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
|
||
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
|
||
tokens to spare. The 2026-10-02 window trim had restored the fit for
|
||
frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
|
||
the fit to frank-dragoon itself. Nothing routes compaction automatically —
|
||
the point is the ceiling no longer forbids it.
|
||
- Two simultaneous generations are the slot ceiling; a third request queues
|
||
behind a released slot (the mechanism measured on the 3-slot shape).
|
||
|
||
## Client contract
|
||
|
||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
|
||
the field `models.yml` reads for this endpoint.
|
||
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
|
||
`enable_thinking` false, so `models.yml` ships
|
||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
|
||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||
- **Vision is on**, through the projector file (`--mmproj`). Use the
|
||
OpenAI-standard `image_url` content part — that path answers correctly
|
||
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
|
||
read by `/v1/chat/completions`: the same request sent that way answered
|
||
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
|
||
but refuses a request without `messages` in this build, so chat is the only
|
||
vision path.
|
||
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-zealot.env` drops
|
||
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
|
||
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
|
||
refusal, never a silent fallback to blindness.
|
||
|
||
## No spec decode
|
||
|
||
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
|
||
ghost), and the cross-model DFlash draft
|
||
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
|
||
optimization candidate, not a default — acceptance has to be measured on this
|
||
traffic first. The bats seam fails if a draft flag ships unannounced.
|
||
|
||
## Rollback
|
||
|
||
|Target|How|
|
||
|---|---|
|
||
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|
||
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|
||
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|
||
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
|
||
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
|
||
|
||
## When it misbehaves
|
||
|
||
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script exits 1
|
||
naming the missing file if the binary or GGUF moved.
|
||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||
- Requests queue behind each other: two concurrent long generations are the
|
||
slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
|
||
free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
|
||
the fourth).
|
||
- `400 … exceeds the available context size (200704 tokens)`: the conversation
|
||
outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
|
||
and the per-slot window is the binding limit. Shorten the prompt, compact,
|
||
or roll back to the 262,144 shape above.
|
||
- Vision answers a color it never saw, or claims the image "is a video":
|
||
either the client sent `images` instead of an `image_url` content part, or
|
||
the probe image is too small to supply real image tokens (use >= 256 px;
|
||
see *Client contract*).
|
||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||
of VRAM into unified memory halves the rate silently.
|