Frank
71bc1735d3
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV accounting, MTP verdict, compaction-hub concurrency), and two Key Lessons bullets -- the draft-context bill restated with the real knob names, plus a new lesson that a quant's scale overhead belongs in its byte rate (the 6.25 % invented by doubling 13,056.00 MiB). LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape; the 3-slot q8_0 boot ledger and its measurements are relabelled as the arm this replaced (byte-identical numbers, honest attribution), the corrected f16 rate derivation is added, the MTP census is labelled q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the boot-ledger section carries an explicit PROJECTION that must be replaced with the measured ledger from the first f16 boot. Pointers re-pointed (live-shape sentences only; every historical ledger left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md, and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k, qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
121 lines
6.2 KiB
Markdown
121 lines
6.2 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||
description: "The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- rtx3090
|
||
- mimo
|
||
- gguf
|
||
- kv-cache
|
||
timestamp: 2026-10-01
|
||
---
|
||
|
||
# frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||
|
||
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
|
||
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||
> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
|
||
> [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork build
|
||
> **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
|
||
> exact f16 K + f16 V, MTP off by measurement).
|
||
>
|
||
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
|
||
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
|
||
> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
|
||
> live endpoint bills **exact f16 K + f16 V** — 32,768 B/token = 16,384 MiB
|
||
> over the 524,288-position pool of two 262,144 windows
|
||
> (`scripts/frank-zealot.env:57-58,80-81`).
|
||
|
||
## Identity
|
||
|
||
|Field|Value|
|
||
|---|---|
|
||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`ALIAS` reached `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09, commit `7c77326`)|
|
||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|
||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|
||
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|
||
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
|
||
|
||
## Shape and the KV contract
|
||
|
||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
|
||
so each session gets `n_ctx_slot = 262144`, the model's full native window
|
||
(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||
|
||
Frank's KV contract: **K unquantized (f16), V `q8_0`** —
|
||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||
|
||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
|
||
path. Without the flag the engine falls back silently and the contract
|
||
becomes a lie. The serve script pins the trio; the bats seam pins the
|
||
script.
|
||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||
(4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
|
||
≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
|
||
**20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
|
||
order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
|
||
K to fix it.
|
||
|
||
## Client contract
|
||
|
||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format
|
||
deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
|
||
`reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
|
||
- **Thinking is off unless the request asks**: the MiMo GGUF template
|
||
defaults `enable_thinking` false; `models.yml` now ships
|
||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
|
||
in `extraBody`. A raw client that omits it gets a no-thinking answer
|
||
(measured — this is not a fault).
|
||
- Tools: `--jinja` + server-side tool parsing, verified
|
||
`tool_calls` round-trip 2026-10-01 23:0x.
|
||
- **Text-only in this era.** The GGUF carries no vision projector; the
|
||
in-checkpoint tower existed only in the SGLang safetensors, so `models.yml`
|
||
carried `input: [text]`. Vision returned with the next payloads: the endpoint
|
||
is `input: [text, image]` today (`~/.omp/agent/models.yml:214`).
|
||
|
||
## No spec decode (yet)
|
||
|
||
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
|
||
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
|
||
Q8_0 — that path is no longer on disk; the 2026-10-18 census of
|
||
`/home/dank/models/` has no `DFlash-Qwen35/`) is an optimization-loop candidate, not a default — `--spec-type
|
||
draft-dflash` needs acceptance measured on this traffic first. The bats
|
||
seam fails if a draft flag ships unannounced.
|
||
|
||
## Build
|
||
|
||
No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
|
||
compiler (nvcc 12.4 rejects gcc ≥ 14):
|
||
|
||
```
|
||
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
|
||
-DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
|
||
-DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
|
||
cmake --build build-cuda -j 16 --target llama-server
|
||
```
|
||
|
||
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) is still on disk; the
|
||
INT8 AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
|
||
is not — it was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`
|
||
on 2026-10-02 and that stage dir is gone as of 2026-10-18, so this rollback
|
||
re-fetches with `old/download-frank-zealot-sglang-int8.sh`: repoint the unit's serve script and env from
|
||
git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
|
||
use the parent of the 2026-10-01 night swap commit).
|
||
|
||
## When it misbehaves
|
||
|
||
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script
|
||
exits 1 naming the missing file if the binary or GGUF moved.
|
||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
|
||
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||
- Requests hang at first token while `/health` is 200: both slots busy —
|
||
two concurrent long generations are the ceiling by contract.
|
||
- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
|
||
VRAM into unified memory halves the rate silently.
|