Frank
06a9cb60eb
Frank's 02:36 instruction (archive 01a14c93) ordered the 27B PTQ1_0 arm at 2 x 200,704 with the Q8 mmproj and batch/ubatch 3072/1536; it served production 03:14:39-03:38:57 (pid 191358, journal type ptq1_0: 402 tensors). Recovered its numbers from that process's own print_timing lines: ~48 t/s single-stream decode, 25-27 t/s per stream at two streams, prefill 1,295 t/s at 30,637 -> 943 at 97,856 -> 801 at 141,365. It bought the promised 1,201 MiB and lost 37% of decode against the PQ2_0 record, so the publisher's Ampere warning generalizes to GA102; Frank scratched the 27B plan at 03:33 for the 9B. Every 'unfinished experiment' claim is re-keyed, and the AGENTS.md law gains the durable lesson: a bench verdict that never reaches the repo is indistinguishable from a rogue arm.
287 lines
17 KiB
Markdown
287 lines
17 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
|
||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
- mtp
|
||
- spec-decode
|
||
timestamp: 2026-10-18
|
||
---
|
||
|
||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
|
||
|
||
Live since 2026-10-18, replacing the
|
||
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
|
||
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
|
||
of the 27B, and that is exactly what funds **three** conversations at the
|
||
model's **native 262,144** window where the 27B afforded only two shortened
|
||
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
|
||
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||
|
||
## Payload identity
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` |
|
||
| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` |
|
||
| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` |
|
||
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) |
|
||
| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id |
|
||
| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` |
|
||
|
||
### The runtime lock has two tumblers now
|
||
|
||
1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's
|
||
PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the
|
||
MTP block) plus `prism.hadamard.*` metadata
|
||
(`normalized-sylvester-walsh-hadamard`, `block_size 4096`,
|
||
`axis input-last-dimension`, `sign_mode explicit` over 201 weight names,
|
||
inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp
|
||
refuses the custom type at load; a loader that ignored the Hadamard metadata
|
||
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
|
||
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
|
||
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
|
||
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
|
||
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||
|
||
## Header census (read from the GGUF, not the card)
|
||
|
||
`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`,
|
||
`general.size_label 9B`, `general.quantization_version 2`,
|
||
`general.file_type 141`:
|
||
|
||
| Key | Value | Note |
|
||
|---|---|---|
|
||
| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block |
|
||
| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers |
|
||
| `qwen35.context_length` | 262,144 | native; the deployed slot equals it |
|
||
| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | |
|
||
| `attention.head_count` / `head_count_kv` | 16 / **4** | |
|
||
| `attention.key_length` / `value_length` | 256 / 256 | |
|
||
| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 |
|
||
| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 |
|
||
| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | |
|
||
| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file |
|
||
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
|
||
|
||
**KV accounting.** Only the 8 full-attention layers scale with context:
|
||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
|
||
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
|
||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
|
||
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
|
||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
|
||
even the recurrent state. Pool arithmetic alone under-prices the shape.
|
||
|
||
## The shape (`scripts/frank-zealot.env`)
|
||
|
||
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
|
||
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
|
||
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
|
||
`--presence-penalty 1.0`,
|
||
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
|
||
|
||
Per-slot window is the **contract** the harness quotes: `models.yml` carries
|
||
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
|
||
400 naming the limit — it never truncates — and since this swap the slot window
|
||
equals the native window, so no YaRN and no cap is in play.
|
||
|
||
## The MTP verdict: the head is in the file and it is still OFF
|
||
|
||
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
|
||
length 2.41, and the card recommends depth 2. It loses **at this shape**
|
||
because the draft context bills its own **f16** KV over the WHOLE pool —
|
||
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
|
||
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
|
||
compute buffers — and `--ubatch-size 2048` refuses to give that up.
|
||
|
||
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
|
||
greedy 512 tokens; never the first request after a boot):
|
||
|
||
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|
||
|---|---|---|---|---|---|---|
|
||
| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** |
|
||
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — |
|
||
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 |
|
||
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — |
|
||
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — |
|
||
|
||
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
|
||
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
|
||
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
|
||
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
|
||
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
|
||
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
|
||
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||
|
||
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
|
||
|
||
```
|
||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||
CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs
|
||
CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output
|
||
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
|
||
nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
|
||
n_slots = 3, n_ctx_slot = 262144, kv_unified = false
|
||
```
|
||
|
||
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||
never the projection**.
|
||
|
||
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
|
||
|
||
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
|
||
(the trial arm printed 167.2; same shape, same noise band).
|
||
- **~300 t/s aggregate** across three concurrent 512-token streams
|
||
(102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
|
||
- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
|
||
has not re-run that depth.
|
||
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
|
||
691.7 prefill at 166k. Beaten on every axis.
|
||
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
|
||
never report it. And never time a wall-clock aggregate on a generation that
|
||
stops on its own: a 24-token countdown measured 72 t/s across three streams
|
||
while the same shape with `ignore_eos` gave 300.
|
||
|
||
## Contract traps
|
||
|
||
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
|
||
`/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
|
||
that is how this endpoint sat red across four payload generations
|
||
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
|
||
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
|
||
llama-server when several image requests run at once; Q8_0 is not affected" —
|
||
and `--parallel 3` is precisely that shape.
|
||
- **Only the `image_url` content part reaches the model.** The legacy top-level
|
||
`images` field bumps `usage.prompt_tokens` and the model still invents an
|
||
answer. Probe with a solid color and expect the exact word.
|
||
- **≥256 px probe images** or the model answers "video".
|
||
- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens
|
||
returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind
|
||
while it is only mid-thought. Budget ≥512.
|
||
- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that
|
||
must reach the argv is this one. The serve script uses
|
||
`${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented
|
||
opt-out and `:=` would silently re-add the flag (caught by the seam, fixed
|
||
2026-10-18).
|
||
- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without
|
||
the whole set has segfaulted at load on this fork family in the past;
|
||
`SPEC_TYPE=none` strips all of them.
|
||
- **Base alignment is the censored base lineage, and the CRACK swap is NOT
|
||
one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B**
|
||
geometry, not this 9B file; a dealigned 9B would have to be fetched.
|
||
- **A raw `/completion` request** logs `failed to parse reasoning content
|
||
(… parse_reasoning=true)` when the model closes a bare prompt with `` —
|
||
cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses
|
||
`reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger
|
||
is what VRAM forensics read.
|
||
- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On
|
||
2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` →
|
||
`Active: failed (Result: timeout)`, while the hand-booted arm kept serving
|
||
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
|
||
unit (llama.cpp needs the drain time to release slots and the card); the
|
||
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
|
||
- **A leftover arm drop-in hijacks the production boot, and no health check
|
||
sees it.** The sanctioned bench method is a transient
|
||
`Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
|
||
`source`s the repo env and then overrides it. On 2026-10-18 a stale
|
||
`frank-zealot.service.d/ptq1.conf` — written 03:29 for the 27B **PTQ1_0** arm
|
||
Frank had ordered at 02:36 and measured from 03:14 (deploy ledger below) —
|
||
was still installed, so the cutover deploy restarted
|
||
the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
|
||
--ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
|
||
slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
|
||
`frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
|
||
passed 33/33 — none of those read the payload. **After deploying an endpoint
|
||
that was recently benched: `systemctl show -p DropInPaths`, then
|
||
`tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
|
||
the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
|
||
|
||
## Compaction hub
|
||
|
||
`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and
|
||
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
|
||
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
|
||
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
|
||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
|
||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||
era's `--parallel 1` law).
|
||
|
||
## Deploy ledger (2026-10-18)
|
||
|
||
- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
|
||
gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
|
||
to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
|
||
ledger recorded here.
|
||
- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
|
||
call fired with schema-correct empty arguments; 512 px solid-red PNG →
|
||
**Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
|
||
journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
|
||
`/metrics` answers 200 with 15 `llamacpp:*` families and
|
||
`up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
|
||
retained history.
|
||
- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
|
||
`webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
|
||
corrected 262,144 → 200,704 — see [[OpenWebUI]]).
|
||
- **What the bench window left behind is not garbage — the PTQ1_0 arm was
|
||
ordered, measured, and then scratched by Frank himself.** The instruction at
|
||
**02:36** was: run frank-zealot on `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, the
|
||
**Q8** mmproj, batch/ubatch **3072/1536**, and as much context as fits in
|
||
**two** contexts. The unit booted it at **03:14** (journal:
|
||
`type ptq1_0: 402 tensors`, `file type = PTQ1_0 - 1.75 bpw ternary`, mmproj
|
||
loaded, 2 × 200,704) and served 24 minutes of traffic. Its own
|
||
`print_timing` lines measure **~48 t/s** single-stream decode
|
||
(48.0 / 47.9 / 47.6 / 49.0 t/s on 512 / 229 / 512 / 451-token generations),
|
||
25–27 t/s per stream with two streams live, and prefill **1,295 t/s at
|
||
30,637 → 943 at 97,856 → 801 t/s at 141,365** tokens. It bought the
|
||
1,201 MiB the fetch script promised and paid for it on decode, exactly as the
|
||
publisher's Ampere row warned (the PQ2_0 record for the same weights is
|
||
76.6 t/s). At **03:33** Frank scratched the whole 27B plan and ordered the
|
||
9B PQ2_0 + MTP shape deployed here; the arm process ended at 03:38.
|
||
- The actual failure was **documentation, not measurement.** An approved
|
||
payload that survives only as a systemd drop-in plus a journal window is,
|
||
a day later, indistinguishable from a rogue arm — which is why the 13:16 boot
|
||
read as a hijack. Payload decisions belong in this runbook the moment they
|
||
are made. The evidence is still perishable: `/tmp/zealot-arm-current.env` and
|
||
the 9B `/tmp/arm-*.jsonl` results die on reboot, and only the journal
|
||
(`journalctl -u frank-zealot --since "2026-10-18 03:10"`) holds the PTQ1_0
|
||
numbers. `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + hardlinked
|
||
mmproj) stays on disk as the 27B rollback.
|
||
|
||
## Fallback rungs
|
||
|
||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||
−17 %).
|
||
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
|
||
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
|
||
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
|
||
last resort, not a knob.
|
||
|
||
## Rollback
|
||
|
||
Revert the swap commit. On disk: the **27B payload**
|
||
(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B
|
||
Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build,
|
||
the Gemma4, MiMo, Swift and Ornith payloads. Then
|
||
`./scripts/deploy.sh dankclaw --service frank-zealot`.
|
||
|
||
## Related
|
||
|
||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target)
|
||
- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method
|
||
- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover
|
||
- [[deployment]] — deploy flow and incidents
|