--- type: reference title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)" description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head." tags: - llm - frank-zealot - dankclaw - llama-cpp - gguf - rtx3090 - mtp - spec-decode timestamp: 2026-10-18 --- # frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV Live since 2026-10-18, replacing the [[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token of the 27B, and that is exactly what funds **three** conversations at the model's **native 262,144** window where the 27B afforded only two shortened ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every axis improved — decode, prefill, three-stream aggregate, and free VRAM. ## Payload identity | Item | Value | |---|---| | Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` | | Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` | | Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` | | Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) | | Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id | | Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` | ### The runtime lock has two tumblers now 1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the MTP block) plus `prism.hadamard.*` metadata (`normalized-sylvester-walsh-hadamard`, `block_size 4096`, `axis input-last-dimension`, `sign_mode explicit` over 201 weight names, inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp refuses the custom type at load; a loader that ignored the Hadamard metadata would dequant **wrong**, silently. Same lock class as the ROCmFPX era. 2. **The build floor.** The fork's card gates **MTP beside a quantized V cache** on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777** for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths. ## Header census (read from the GGUF, not the card) `general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`, `general.size_label 9B`, `general.quantization_version 2`, `general.file_type 141`: | Key | Value | Note | |---|---|---| | `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block | | `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers | | `qwen35.context_length` | 262,144 | native; the deployed slot equals it | | `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | | | `attention.head_count` / `head_count_kv` | 16 / **4** | | | `attention.key_length` / `value_length` | 256 / 256 | | | `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 | | `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 | | `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | | | `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file | | `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself | **KV accounting.** Only the 8 full-attention layers scale with context: 8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so **q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is **13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token. **f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at `cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed 452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even the recurrent state. Pool arithmetic alone under-prices the shape. ## The shape (`scripts/frank-zealot.env`) `--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`, `--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`, `--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`, `--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`, `--presence-penalty 1.0`, **`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`). Per-slot window is the **contract** the harness quotes: `models.yml` carries `contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a 400 naming the limit — it never truncates — and since this swap the slot window equals the native window, so no YaRN and no cap is in play. ## The MTP verdict: the head is in the file and it is still OFF The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the card recommends depth 2. It loses **at this shape** because the draft context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its compute buffers — and `--ubatch-size 2048` refuses to give that up. Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`, greedy 512 tokens; never the first request after a boot): | slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k | |---|---|---|---|---|---|---| | **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** | | 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — | | 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 | | 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — | | 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — | **MTP is worth +28 % single-stream decode whenever the card has room** (rows 4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 % prefill and +12 % three-stream aggregate over single-stream speed, and keeps 6.2 GiB free. Frank confirmed this arm on 2026-10-18. ## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4) ``` CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap) CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123 n_slots = 3, n_ctx_slot = 262144, kv_unified = false ``` No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`, never the projection**. ## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`) - **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7 (the trial arm printed 167.2; same shape, same noise band). - **~300 t/s aggregate** across three concurrent 512-token streams (102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2. - **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot has not re-run that depth. - The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at 166k. Beaten on every axis. - The **first request after a boot is a warmup artifact** (37.7 t/s class) — never report it. And never time a wall-clock aggregate on a generation that stops on its own: a 24-token countdown measured 72 t/s across three streams while the same shape with `ignore_eos` gave 300. ## Contract traps - **`--metrics` is mandatory.** Without it llama.cpp answers **501** on `/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 — that is how this endpoint sat red across four payload generations (`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0). - **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash llama-server when several image requests run at once; Q8_0 is not affected" — and `--parallel 3` is precisely that shape. - **Only the `image_url` content part reaches the model.** The legacy top-level `images` field bumps `usage.prompt_tokens` and the model still invents an answer. Probe with a solid color and expect the exact word. - **≥256 px probe images** or the model answers "video". - **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind while it is only mid-thought. Budget ≥512. - **`presence_penalty` has no GGUF metadata key**, so the one recipe value that must reach the argv is this one. The serve script uses `${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented opt-out and `:=` would silently re-add the flag (caught by the seam, fixed 2026-10-18). - **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without the whole set has segfaulted at load on this fork family in the past; `SPEC_TYPE=none` strips all of them. - **Base alignment is the censored base lineage, and the CRACK swap is NOT one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry, not this 9B file; a dealigned 9B would have to be fetched. - **A raw `/completion` request** logs `failed to parse reasoning content (… parse_reasoning=true)` when the model closes a bare prompt with `` — cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses `reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger is what VRAM forensics read. - **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On 2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` → `Active: failed (Result: timeout)`, while the hand-booted arm kept serving 7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the unit (llama.cpp needs the drain time to release slots and the card); the hand-booted arm must be killed by PID, never `pkill -f llama-server`. - **A leftover arm drop-in hijacks the production boot, and no health check sees it.** The sanctioned bench method is a transient `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/.env` drop-in, and an arm env `source`s the repo env and then overrides it. On 2026-10-18 a stale `frank-zealot.service.d/ptq1.conf` — written 03:29 for the 27B **PTQ1_0** arm Frank had ordered at 02:36 and measured from 03:14 (deploy ledger below) — was still installed, so the cutover deploy restarted the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf --ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said `frank-zealot` (the alias is a flag, not a fact), and the Playwright gate passed 33/33 — none of those read the payload. **After deploying an endpoint that was recently benched: `systemctl show -p DropInPaths`, then `tr '\0' ' ' -/`, `daemon-reload`, restart. ## Compaction hub `models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and `frank-zealot` here — 262,144 is the widest window in the fleet and the only one that comfortably fits a worst-case compaction input (~170k tokens; the clamp is `min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3` a compaction request no longer serializes behind a live generation (the Gemma4 era's `--parallel 1` law). ## Deploy ledger (2026-10-18) - `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e gate ran 33/33. The first boot was the hijacked arm above; the drop-in went to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the ledger recorded here. - Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool call fired with schema-correct empty arguments; 512 px solid-red PNG → **Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`; `/metrics` answers 200 with 15 `llamacpp:*` families and `up{job="frank-zealot"}` is **1** in Prometheus for the first time in its retained history. - OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot `webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also corrected 262,144 → 200,704 — see [[OpenWebUI]]). - **What the bench window left behind is not garbage — the PTQ1_0 arm was ordered, measured, and then scratched by Frank himself.** The instruction at **02:36** was: run frank-zealot on `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, the **Q8** mmproj, batch/ubatch **3072/1536**, and as much context as fits in **two** contexts. The unit booted it at **03:14** (journal: `type ptq1_0: 402 tensors`, `file type = PTQ1_0 - 1.75 bpw ternary`, mmproj loaded, 2 × 200,704) and served 24 minutes of traffic. Its own `print_timing` lines measure **~48 t/s** single-stream decode (48.0 / 47.9 / 47.6 / 49.0 t/s on 512 / 229 / 512 / 451-token generations), 25–27 t/s per stream with two streams live, and prefill **1,295 t/s at 30,637 → 943 at 97,856 → 801 t/s at 141,365** tokens. It bought the 1,201 MiB the fetch script promised and paid for it on decode, exactly as the publisher's Ampere row warned (the PQ2_0 record for the same weights is 76.6 t/s). At **03:33** Frank scratched the whole 27B plan and ordered the 9B PQ2_0 + MTP shape deployed here; the arm process ended at 03:38. - The actual failure was **documentation, not measurement.** An approved payload that survives only as a systemd drop-in plus a journal window is, a day later, indistinguishable from a rogue arm — which is why the 13:16 boot read as a hijack. Payload decisions belong in this runbook the moment they are made. The evidence is still perishable: `/tmp/zealot-arm-current.env` and the 9B `/tmp/arm-*.jsonl` results die on reboot, and only the journal (`journalctl -u frank-zealot --since "2026-10-18 03:10"`) holds the PTQ1_0 numbers. `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + hardlinked mmproj) stays on disk as the 27B rollback. ## Fallback rungs 1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill −17 %). 2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704). 3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a last resort, not a knob. ## Rollback Revert the swap commit. On disk: the **27B payload** (`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build, the Gemma4, MiMo, Swift and Ornith payloads. Then `./scripts/deploy.sh dankclaw --service frank-zealot`. ## Related - [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target) - [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method - [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover - [[deployment]] — deploy flow and incidents