serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)	The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head.	llm frank-zealot dankclaw llama-cpp gguf rtx3090 mtp spec-decode	2026-10-18

frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV

Live since 2026-10-18, replacing the frank-zealot-llama-prism-bonsai2-pq20 era (Ternary-Bonsai-2-27B PQ2_0 at 2 × 200,704). The trade in one sentence: the 9B bills half the KV per token of the 27B, and that is exactly what funds three conversations at the model's native 262,144 window where the 27B afforded only two shortened ones (Frank's ask: "if you can possibly fit a third context, do it"). Every axis improved — decode, prefill, three-stream aggregate, and free VRAM.

Payload identity

Item	Value
Runtime	/home/dank/GitPackages/llama-prism-b10777/llama-server — PrismML fork build, --version → 0.2.0-dev (build 10777, commit e3de71522)
Checkpoint	/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf — 2,675,410,720 B, Hub-LFS sha256 c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b
Vision	/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf — 624,229,664 B, Hub-LFS sha256 e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db
Fetch	scripts/download-frank-zealot-bonsai9.sh [DEST] (--verify-only; -f, byte-count and LFS sha256 per file)
Alias	frank-zealot (ALIAS); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id
Port	7771, root systemd unit frank-zealot.service (relinked), TimeoutStopSec=180

The runtime lock has two tumblers now

1. The tensor type. general.file_type 141 with 202 tensors of the fork's PQ2_0 type (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the MTP block) plus prism.hadamard.* metadata (normalized-sylvester-walsh-hadamard, block_size 4096, axis input-last-dimension, sign_mode explicit over 201 weight names, inverse on token_embd.weight, gdn_v_grouped true). Stock llama.cpp refuses the custom type at load; a loader that ignored the Hadamard metadata would dequant wrong, silently. Same lock class as the ROCmFPX era.
2. The build floor. The fork's card gates MTP beside a quantized V cache on prism-b10776-d00e252 or newer, so the prebuilt moved 10754 → 10777 for this swap. The 10754 tree stays on disk (~/GitPackages/llama-prism/) as the 27B rollback runtime. tests/frank_zealot_unit.bats pins both paths.

Header census (read from the GGUF, not the card)

general.architecture qwen35, general.name Ternary-Bonsai-2-9B, general.size_label 9B, general.quantization_version 2, general.file_type 141:

Key	Value	Note
qwen35.block_count	33	32 trunk blocks + blk.32 = the MTP block
qwen35.full_attention_interval	4	⇒ 8 full-attention + 24 Gated DeltaNet trunk layers
qwen35.context_length	262,144	native; the deployed slot equals it
qwen35.embedding_length / feed_forward_length	4,096 / 12,288
attention.head_count / head_count_kv	16 / 4
attention.key_length / value_length	256 / 256
ssm.conv_kernel / state_size / group_count	4 / 128 / 16	the 27B: 4 / 128 / 16
ssm.time_step_rank / inner_size	32 / 4,096	the 27B: 48 / 6,144
rope.freq_base / dimension_count / dimension_sections	1e7 / 64 / [11,11,10,0]
qwen35.nextn_predict_layers	1	the MTP head is IN the file
general.sampling.*	temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0	the card recipe; llama.cpp reads it itself

KV accounting. Only the 8 full-attention layers scale with context: 8 × 4 KV heads × 256 dim = 8,192 elements/token per cache, so q8_0 K + q8_0 V bills 17,408 B/token — the 786,432-position pool is 13,056.00 MiB, ledger-exact. That is exactly HALF the 27B's 34,816 B/token. f16 KV would bill 26,112 MiB against a 24,123 MiB card and dies at cudaMalloc. The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead — RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs) ≈ 150 MiB each — so pool arithmetic alone under-prices the shape.

The shape (scripts/frank-zealot.env)

--ctx-size 786432 --parallel 3 (⇒ n_ctx_slot = 262,144), --flash-attn on, --cache-type-k q8_0 --cache-type-v q8_0, --batch-size 4096, --ubatch-size 2048, --n-gpu-layers 99, --jinja, --reasoning auto, --reasoning-format deepseek, --metrics, --mmproj …, --log-verbosity 4, --presence-penalty 1.0, SPEC_TYPE=none (which strips --spec-type AND every --spec-draft-*).

Per-slot window is the contract the harness quotes: models.yml carries contextWindow: 262144. llama.cpp refuses a prompt longer than the SLOT with a 400 naming the limit — it never truncates — and since this swap the slot window equals the native window, so no YaRN and no cap is in play.

The MTP verdict: the head is in the file and it is still OFF

The drafter is good: acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the card recommends depth 2. It loses at this shape because the draft context bills its own f16 KV over the WHOLE pool — blk.32 carries real attention (4 KV heads × 256 × 2 caches × 2 B = 4,096 B/position) × 786,432 positions = 3,072.00 MiB, 4,830 MiB with its compute buffers — and --ubatch-size 2048 refuses to give that up.

Booted arms, same card, same traffic, 2026-10-18 (/completion timings, greedy 512 tokens; never the first request after a boot):

slots	spec	ubatch	used / free	decode	3-stream	prefill @106k
3 × 262,144	off	2048	17,876 / 6,249	167.2	280.2	2,877
3 × 262,144	on	2048	22,706 / 1,419	109.2	—	—
3 × 262,144	on	1024	21,400 / 2,725	213.5	250.8	2,395
2 × 262,144	on	2048	17,178 / 6,947	213.2	—	—
1 × 262,144	on	2048	11,652 / 12,473	214.2	—	—

MTP is worth +28 % single-stream decode whenever the card has room (rows 4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves it). The arm that runs MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (row 3) and pays prefill −17 % and 2,725 MiB of headroom on a card whose mmproj worst-case lands AFTER the engine's own fit check. This endpoint is the fleet's compaction hub — prefill-bound ~170k inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 % prefill and +12 % three-stream aggregate over single-stream speed, and keeps 6.2 GiB free. Frank confirmed this arm on 2026-10-18.

Boot ledger (deployed arm, verbosity 4)

CUDA0 KV buffer size = 13056.00 MiB        # q8_0 K + q8_0 V, 786,432 positions
CUDA0 RS buffer size =   452.25 MiB        # 3 recurrent-state cells, 32 layers, 3 seqs
compute   1,409 MiB CUDA + 1,201 MiB CUDA_Host
mmproj      595 MiB + 248 MiB              # charged AFTER the fit check
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123

No clamp line, no cudaMalloc error. The engine's own projection (21,439-class numbers from the 27B era) excludes the projector — read settled nvidia-smi, never the projection.

Measured (deployed arm)

• 167.2 t/s greedy decode (512 tokens, /completion timings); 158.9 t/s single-stream chat with the card's sampler.
• 280.2 t/s aggregate across three concurrent streams (≈93.8 each).
• 2,877 t/s prefill at 106,418 prompt tokens.
• The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at 166k. Beaten on every axis.
• The first request after a boot is a warmup artifact (37.7 t/s class) — never report it.

Contract traps

• --metrics is mandatory. Without it llama.cpp answers 501 on /metrics, so the Prometheus scrape reads DOWN while /health answers 200 — that is how this endpoint sat red across four payload generations (up{job="frank-zealot"} oldest retained sample, 2026-10-11, is already 0).
• Q8_0 mmproj, not BF16. The publisher's card: the BF16 tower "can crash llama-server when several image requests run at once; Q8_0 is not affected" — and --parallel 3 is precisely that shape.
• Only the image_url content part reaches the model. The legacy top-level images field bumps usage.prompt_tokens and the model still invents an answer. Probe with a solid color and expect the exact word.
• ≥256 px probe images or the model answers "video".
• Thinking tokens count against max_tokens. A vision probe at 64 tokens returns EMPTY content + finish_reason: "length" — the endpoint looks blind while it is only mid-thought. Budget ≥512.
• presence_penalty has no GGUF metadata key, so the one recipe value that must reach the argv is this one. The serve script uses ${PRESENCE_PENALTY=1.0} — not := — because "" is the documented opt-out and := would silently re-add the flag (caught by the seam, fixed 2026-10-18).
• Spec flags move as a SET. A bare --model-draft/--spec-type arm without the whole set has segfaulted at load on this fork family in the past; SPEC_TYPE=none strips all of them.
• Base alignment is the censored base lineage, and the CRACK swap is NOT one line here. /home/dank/models/Bonsai2-CRACK/ matches the 27B geometry, not this 9B file; a dealigned 9B would have to be fetched.
• A raw /completion request logs failed to parse reasoning content (… parse_reasoning=true) when the model closes a bare prompt with `` — cosmetic (probe_request.cpp:789); /v1/chat/completions parses reasoning_content correctly. Do not lower VERBOSITY: the KV/buffer ledger is what VRAM forensics read.
• A failed stop leaves a failed unit, and an orphan keeps answering. On 2026-10-18 the trial stop hit State 'stop-sigterm' timed out. Killing. → Active: failed (Result: timeout), while the hand-booted arm kept serving 7771 — a live port with no unit behind it. TimeoutStopSec=180 is now on the unit (llama.cpp needs the drain time to release slots and the card); the hand-booted arm must be killed by PID, never pkill -f llama-server.

Compaction hub

models.yml routes compactionModel for frank-archon, frank-dragoon and frank-zealot here — 262,144 is the widest window in the fleet and the only one that comfortably fits a worst-case compaction input (~170k tokens; the clamp is min(maxTokens 16384, contextWindow − est − 4096), compaction fires at 85 % of the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With --parallel 3 a compaction request no longer serializes behind a live generation (the Gemma4 era's --parallel 1 law).

Fallback rungs

1. UBATCH_SIZE=1024 — frees 1,306 MiB measured on this shape (prefill −17 %).
2. CACHE_TYPE_K/V=q4_0 — frees ~6.5 GiB at three windows (the 27B era measured the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a last resort, not a knob.

Rollback

Revert the swap commit. On disk: the 27B payload (/home/dank/models/Bonsai2-PQ2_0/) + the 10754 fork tree, the Qwen3.5-9B Q6_K payload (/home/dank/models/Qwen35-9B-HauhauCS-Q6K/) + master CUDA build, the Gemma4, MiMo, Swift and Ornith payloads. Then ./scripts/deploy.sh dankclaw --service frank-zealot.

Related

• frank-zealot-llama-prism-bonsai2-pq20 — the retired 27B era (rollback target)
• Llama Cpp Serving Sizing — the two-boot VRAM pricing method
• frank-dragoon-flash-server-v2-native — the other 2026-10-18 cutover
• deployment — deploy flow and incidents