Frank
466fd477d8
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
223 lines
12 KiB
Markdown
223 lines
12 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
|
||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
- mtp
|
||
- spec-decode
|
||
timestamp: 2026-10-18
|
||
---
|
||
|
||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
|
||
|
||
Live since 2026-10-18, replacing the
|
||
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
|
||
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
|
||
of the 27B, and that is exactly what funds **three** conversations at the
|
||
model's **native 262,144** window where the 27B afforded only two shortened
|
||
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
|
||
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||
|
||
## Payload identity
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` |
|
||
| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` |
|
||
| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` |
|
||
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) |
|
||
| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id |
|
||
| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` |
|
||
|
||
### The runtime lock has two tumblers now
|
||
|
||
1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's
|
||
PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the
|
||
MTP block) plus `prism.hadamard.*` metadata
|
||
(`normalized-sylvester-walsh-hadamard`, `block_size 4096`,
|
||
`axis input-last-dimension`, `sign_mode explicit` over 201 weight names,
|
||
inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp
|
||
refuses the custom type at load; a loader that ignored the Hadamard metadata
|
||
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
|
||
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
|
||
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
|
||
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
|
||
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||
|
||
## Header census (read from the GGUF, not the card)
|
||
|
||
`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`,
|
||
`general.size_label 9B`, `general.quantization_version 2`,
|
||
`general.file_type 141`:
|
||
|
||
| Key | Value | Note |
|
||
|---|---|---|
|
||
| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block |
|
||
| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers |
|
||
| `qwen35.context_length` | 262,144 | native; the deployed slot equals it |
|
||
| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | |
|
||
| `attention.head_count` / `head_count_kv` | 16 / **4** | |
|
||
| `attention.key_length` / `value_length` | 256 / 256 | |
|
||
| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 |
|
||
| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 |
|
||
| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | |
|
||
| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file |
|
||
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
|
||
|
||
**KV accounting.** Only the 8 full-attention layers scale with context:
|
||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
|
||
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
|
||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||
instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB
|
||
each — so pool arithmetic alone under-prices the shape.
|
||
|
||
## The shape (`scripts/frank-zealot.env`)
|
||
|
||
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
|
||
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
|
||
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
|
||
`--presence-penalty 1.0`,
|
||
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
|
||
|
||
Per-slot window is the **contract** the harness quotes: `models.yml` carries
|
||
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
|
||
400 naming the limit — it never truncates — and since this swap the slot window
|
||
equals the native window, so no YaRN and no cap is in play.
|
||
|
||
## The MTP verdict: the head is in the file and it is still OFF
|
||
|
||
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
|
||
length 2.41, and the card recommends depth 2. It loses **at this shape**
|
||
because the draft context bills its own **f16** KV over the WHOLE pool —
|
||
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
|
||
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
|
||
compute buffers — and `--ubatch-size 2048` refuses to give that up.
|
||
|
||
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
|
||
greedy 512 tokens; never the first request after a boot):
|
||
|
||
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|
||
|---|---|---|---|---|---|---|
|
||
| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** |
|
||
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — |
|
||
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 |
|
||
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — |
|
||
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — |
|
||
|
||
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
|
||
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
|
||
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
|
||
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
|
||
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
|
||
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
|
||
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||
|
||
## Boot ledger (deployed arm, verbosity 4)
|
||
|
||
```
|
||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||
CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs
|
||
compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host
|
||
mmproj 595 MiB + 248 MiB # charged AFTER the fit check
|
||
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123
|
||
```
|
||
|
||
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||
never the projection**.
|
||
|
||
## Measured (deployed arm)
|
||
|
||
- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s
|
||
single-stream chat with the card's sampler.
|
||
- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each).
|
||
- **2,877 t/s prefill** at 106,418 prompt tokens.
|
||
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
|
||
691.7 prefill at 166k. Beaten on every axis.
|
||
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
|
||
never report it.
|
||
|
||
## Contract traps
|
||
|
||
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
|
||
`/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
|
||
that is how this endpoint sat red across four payload generations
|
||
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
|
||
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
|
||
llama-server when several image requests run at once; Q8_0 is not affected" —
|
||
and `--parallel 3` is precisely that shape.
|
||
- **Only the `image_url` content part reaches the model.** The legacy top-level
|
||
`images` field bumps `usage.prompt_tokens` and the model still invents an
|
||
answer. Probe with a solid color and expect the exact word.
|
||
- **≥256 px probe images** or the model answers "video".
|
||
- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens
|
||
returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind
|
||
while it is only mid-thought. Budget ≥512.
|
||
- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that
|
||
must reach the argv is this one. The serve script uses
|
||
`${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented
|
||
opt-out and `:=` would silently re-add the flag (caught by the seam, fixed
|
||
2026-10-18).
|
||
- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without
|
||
the whole set has segfaulted at load on this fork family in the past;
|
||
`SPEC_TYPE=none` strips all of them.
|
||
- **Base alignment is the censored base lineage, and the CRACK swap is NOT
|
||
one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B**
|
||
geometry, not this 9B file; a dealigned 9B would have to be fetched.
|
||
- **A raw `/completion` request** logs `failed to parse reasoning content
|
||
(… parse_reasoning=true)` when the model closes a bare prompt with `` —
|
||
cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses
|
||
`reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger
|
||
is what VRAM forensics read.
|
||
- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On
|
||
2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` →
|
||
`Active: failed (Result: timeout)`, while the hand-booted arm kept serving
|
||
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
|
||
unit (llama.cpp needs the drain time to release slots and the card); the
|
||
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
|
||
|
||
## Compaction hub
|
||
|
||
`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and
|
||
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
|
||
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
|
||
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
|
||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
|
||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||
era's `--parallel 1` law).
|
||
|
||
## Fallback rungs
|
||
|
||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||
−17 %).
|
||
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
|
||
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
|
||
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
|
||
last resort, not a knob.
|
||
|
||
## Rollback
|
||
|
||
Revert the swap commit. On disk: the **27B payload**
|
||
(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B
|
||
Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build,
|
||
the Gemma4, MiMo, Swift and Ornith payloads. Then
|
||
`./scripts/deploy.sh dankclaw --service frank-zealot`.
|
||
|
||
## Related
|
||
|
||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target)
|
||
- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method
|
||
- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover
|
||
- [[deployment]] — deploy flow and incidents
|