serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)	The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape.	llm frank-zealot dankclaw llama-cpp gguf rtx3090	2026-10-17

frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV

Live since 2026-10-17, replacing the frank-zealot-llama-qwen35-9b-q6k-2x200704 era (Qwen3.5-9B Q6_K, exact f16 KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights of the 9B and buys a 3× larger, stronger base at the SAME window contract — paid for by quantizing KV (f16 KV is arithmetically impossible for this payload on this card, measured below).

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine)
Endpoint	http://192.168.0.143:7771 — served model id frank-zealot (ALIAS), /v1 OpenAI surface
Runtime	PrismML fork build /home/dank/GitPackages/llama-prism/llama-server — vendor release, --version = 0.2.0-dev (build 10754, commit 2459f68b5), sha256 12ae5baba033d8cd…. NOT a preference: see Why the fork
Unit	root systemd frank-zealot.service (scripts/frank-zealot.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-zealot.sh, tunables scripts/frank-zealot.env
Weights	/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf — 7,206,168,928 B, sha256 3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1
Projector	Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; 873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check
Fetch	scripts/download-frank-zealot-bonsai2.sh — -f, byte-count + sha256 verified, idempotent, --verify-only, no token on the command line
Seam	tests/frank_zealot_unit.bats (argv assertions against a stub engine)

Why the fork — the format is the lock

Read from the file's own metadata, not the model card:

Key	Value
general.architecture	qwen35 (hybrid: Gated DeltaNet + softmax attention)
qwen35.block_count	64, full_attention_interval 4 ⇒ 16 full-attention + 48 DeltaNet layers
Attention	24 heads / 4 KV heads × key 256 / value 256; rope.dimension_sections [11,11,10,0], freq_base 1e7, partial rope dimension_count 64
Widths	embedding 5,120, feed-forward 17,408
qwen35.ssm.*	conv_kernel 4, state_size 128, group_count 16, time_step_rank 48, inner_size 6,144
Window / vocab	context_length 262,144; vocab 248,320 (gpt2 BPE, pre qwen35), eos 248046
general.file_type	141 = PQ2_0 — 2.13 bpw (group 128)
Tensor histogram	851 tensors: pq2_0 ×402 (custom GGML type 142), f32 ×353, bf16 ×96
prism.hadamard.*	transform normalized-sylvester-walsh-hadamard, block_size 1024, axis input-last-dimension, sign_mode explicit, 401 named weights, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on token_embd.weight, gdn_v_grouped true
Sampler metadata	general.sampling.temp 1.0 / top_p 0.95 / top_k 20
Chat template	8,952 chars; reads enable_thinking, reasoning_effort ∈ {xhigh (default), medium, low — anything else is raise_exception, preserve_thinking, tools/tool_call, image + video content parts

Stock llama.cpp cannot load this file: type 142 is not in its enum (gguf-py's stock GGMLQuantizationType raises ValueError: 142 is not a valid… on the same read), and even a loader that parsed it would ignore the prism.hadamard.* dequant transform the weights depend on. Same class of lock as the ROCmFPX custom-type era. The vendor build 10754 already carries the September upstream fixes (AVX-512 load segfault, and a GGML_CUDA_FA_ALL_QUANTS -capable CUDA backend — proven by q8_0 V serving under flash attention at the correct 34,816 B/token, below).

VRAM census (2026-10-17, four arms, production stopped)

The engine's own common_memory_breakdown_print ledger + settled nvidia-smi, all arms --n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048, mmproj attached:

Shape	KV	projected model + context + compute	settled used / free	verdict
2 × 200,704 (pool 401,408)	q8_0 / q8_0	21,439 = 6,539 + 13,627 + 1,272	22,664 / 1,461	DEPLOYED
2 × 200,704	f16 / f16	33,198 = 6,539 + 25,387 + 1,272	—	cudaMalloc failed: out of memory on the 25,088.00 MiB KV request
1 × 262,144	q8_0 / q8_0	16,905 = 6,539 + 8,853 + 1,512	18,130 / 5,995	fits, but one slot serializes compaction
2 × 262,144	q8_0 / q8_0	≈25,758 (arithmetic, same terms)	—	does not fit (f16 analogue measured 41,118)
2 × 200,704	q4_0 / q4_0	15,167 = 6,539 + 7,355 + 1,272	16,392 / 7,733	the roomy fallback rung

KV rate law (confirmed to the MiB): 16 full layers × 4 KV heads × 256 dim = 16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ 34,816 B/token, so the 401,408 pool = 13,328.00 MiB — the ledger printed exactly that (CUDA0 KV buffer size = 13328.00 MiB). f16 doubles it to 25,088 MiB, which is the wall. The 48 DeltaNet layers bill a fixed recurrent state per slot: context 13,627 − KV 13,328 = 299 MiB for two slots (~150 MiB each) — pool arithmetic alone under-prices the shape by that much.

The projector is outside the fit check. [mtmd] estimated worst-case memory usage of mmproj is 873.10 MiB is printed before fitting but not billed in it: settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the CUDA context. This is the difference between "2.6 GiB free on paper" and the real 1.4 GiB — the same late-allocation failure class that killed the DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled nvidia-smi, never the projection.

Measured rates (greedy, Server-Timing-free /completion timings)

Probe	Result
Short, warm, single stream (128 tok)	76.6 t/s decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)
First request after boot	2.6 t/s prefill / 37.7 t/s decode — warmup artifact; never report the first request (measured twice on this payload)
Prefill at ~35k depth	1,258.4 t/s (27.8 s)
Prefill at 166,027-token depth	691.7 t/s (240.0 s), zero errors, slot intact
Two concurrent streams, 256 tok each	54.7 + 54.7 = 109.4 t/s aggregate — --parallel 2 really batches
Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)	exact: "Frank keeps the fleet in git, only through it."

Production boot (the 19:44 cutover deploy, same binary/flags/file): 75.8 t/s single-stream decode at a short prompt — within 1 % of the trial arm, so the ladder above carries to the shipped shape.

Behavior contract (probes, not assumptions)

• Reasoning lands in reasoning_content (--reasoning-format deepseek) ✓.
• Tool calls: a get_weather request returns finish_reason: tool_calls with arguments: {"city":"Seattle"} — valid JSON, no loop, no HTTP 500. The fork's KNOWN_ISSUES.md warning about malformed/looping tool calls did not reproduce on build 10754 at this shape.
• Vision: a 512 px solid-red PNG answers "Red" — but ONLY with a thinking budget ≥ ~512 tokens. At max_tokens: 64 the request returns finish_reason: "length" with EMPTY content and half a thought in reasoning_content: thinking tokens count against the budget, so a short-budget vision probe reads as "the model is blind". Images do reach the model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text alone). Keep the ≥256 px probe rule from the earlier eras.
• Log noise law: /completion (text context) at VERBOSITY=4 prints failed to parse reasoning content (response_format json_schema is currently only supported with parse_reasoning=true), content will be stored in 'content' field (probe_request.cpp:789) whenever the model closes a raw prompt with ``. It is cosmetic — the full chain of thought is in content. The OpenAI /v1/chat/completions path parses it into reasoning_content correctly, which is the path the harness and every consumer uses. Do not diagnose this as a fault, and do not drop verbosity below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
• Long raw-prompt completions can come back empty: at ~26k+ tokens a bare /completion request generated 0–1 tokens with no error (the model emits an end-of-turn marker immediately, and a text context has no role frame to keep it talking). Chat traffic is unaffected; nothing in the harness posts raw /completion at depth.

Ops notes

• Trial-window law (cost one boot window on 2026-10-17): a scratch env file must use the serve script's OWN variable names — CACHE_TYPE_K/CACHE_TYPE_V, BIND_ADDR, NGL. An invented KV_CACHE_TYPE is silently swallowed by the script's : "${CACHE_TYPE_K:=f16}" default and the arm boots f16, which for this payload is an OOM that looks like a model problem.
• Base alignment: Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a refusal is possible on a compaction or subagent request. The CRACK re-export (dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF, already on disk at /home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf, sha256 5b24ea3e…) is a one-line MODEL_FILE swap: the header census proves byte-identical geometry — 851 tensors, the same {142:402, f32:353, bf16:96} histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM ledger does not move. If a refusal ever surfaces, that is the fix; there is also an MTP-bearing CRACK re-export upstream (ManniX-ITA/…-CRACK-MTP-GGUF, 7,657,489,920 B, +451 MB head) if decode ever needs the drafter — untested here.
• Compaction hub unchanged: all three endpoints still route compactionModel here; 200,704 per slot is the fleet's only window that fits a worst-case compaction input, and --parallel 2 keeps compaction off the critical path of a live generation.
• Harness side (~/.omp/agent/models.yml): contextWindow: 200704 and maxTokens: 16384 unchanged; extraBody now quotes the payload's own 1.0 / 0.95 / 20 with preserve_thinking: true. The reasoningEffortMap fold became load-bearing (the template raises on any level outside xhigh|medium|low) — an unmapped level is a hard 500 on the role's first request, naming Jinja rather than the model.
• Rollback: revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj and the master CUDA build all remain on disk (/home/dank/models/Qwen35-9B-HauhauCS-Q6K/, ~/GitPackages/llama-master/build-cuda/), and the retired era keeps its own page: frank-zealot-llama-qwen35-9b-q6k-2x200704.
• Deployed 2026-10-17 19:44 (deploy.sh dankclaw --service frank-zealot --service homepage): boot ledger 22,664 used / 1,461 free, engine fit projected 21,439 and left 2,119 ≥ 1,872 MiB, KV 13,328.00 MiB at n_ctx_seq = 200,704, n_slots = 2, projector loaded; live probes green (/v1/models serves frank-zealot, exact pong + reasoning_content, tool_calls {"city":"Seattle"}, 512 px PNG → Red). e2e gate 33/33.
• Picker row is live state: the OpenWebUI model row's meta.description named the 9B, so it was re-keyed in the database too (backup webui.db.bak-20261017a). The row was written WITHOUT restarting open-webui — the cached list refreshes on its next restart, and forcing one would have logged every family session out for a cosmetic string. See OpenWebUI for the picker trio.

Related

• frank-zealot-llama-qwen35-9b-q6k-2x200704 — the replaced era (rollback target)
• frank-dragoon-flash-server-v2-swift15 — the abliterated sibling on the iGPU
• frank-archon-radiance-qwen38-iq4r — the censored-base flagship this payload shares base alignment with
• deployment — deploy flow and incidents