Frank
4607604ef7
Left uncommitted by the deploy session: the production decode rate (75.8 t/s, within 1% of the trial arm), the shipped boot ledger, the live probe census, and the picker-row law (OpenWebUI meta.description is live DB state a swap must follow into the database, written without restarting open-webui).
191 lines
13 KiB
Markdown
191 lines
13 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)"
|
||
description: "The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
timestamp: 2026-10-17
|
||
---
|
||
|
||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
|
||
|
||
Live since 2026-10-17, replacing the
|
||
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] era (Qwen3.5-9B Q6_K, exact f16
|
||
KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights
|
||
of the 9B and buys a 3× larger, stronger base at the SAME window contract —
|
||
paid for by quantizing KV (f16 KV is arithmetically impossible for this
|
||
payload on this card, measured below).
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
|
||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
|
||
| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
|
||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||
| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
|
||
| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
|
||
| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
|
||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
|
||
|
||
## Why the fork — the format is the lock
|
||
|
||
Read from the file's own metadata, not the model card:
|
||
|
||
|Key|Value|
|
||
|---|---|
|
||
|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
|
||
|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
|
||
|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
|
||
|Widths|embedding 5,120, feed-forward 17,408|
|
||
|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
|
||
|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
|
||
|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
|
||
|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
|
||
|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
|
||
|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
|
||
|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
|
||
|
||
Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
|
||
stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
|
||
same read), and even a loader that parsed it would ignore the
|
||
`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
|
||
as the ROCmFPX custom-type era. The vendor build 10754 already carries the
|
||
September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
|
||
-capable CUDA backend — proven by q8_0 V serving under flash attention at the
|
||
correct 34,816 B/token, below).
|
||
|
||
## VRAM census (2026-10-17, four arms, production stopped)
|
||
|
||
The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
|
||
all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
|
||
mmproj attached:
|
||
|
||
|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
|
||
|---|---|---|---|---|
|
||
|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
|
||
|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
|
||
|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
|
||
|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
|
||
|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
|
||
|
||
**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
|
||
16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
|
||
so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
|
||
(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
|
||
the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
|
||
context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
|
||
arithmetic alone under-prices the shape by that much.
|
||
|
||
**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
|
||
usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
|
||
settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
|
||
CUDA context. This is the difference between "2.6 GiB free on paper" and the
|
||
real **1.4 GiB** — the same late-allocation failure class that killed the
|
||
DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
|
||
`nvidia-smi`, never the projection.
|
||
|
||
## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
|
||
|
||
|Probe|Result|
|
||
|---|---|
|
||
|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
|
||
|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
|
||
|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
|
||
|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
|
||
|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
|
||
|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
|
||
|
||
Production boot (the 19:44 cutover deploy, same binary/flags/file): **75.8 t/s**
|
||
single-stream decode at a short prompt — within 1 % of the trial arm, so the
|
||
ladder above carries to the shipped shape.
|
||
|
||
## Behavior contract (probes, not assumptions)
|
||
|
||
- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
|
||
- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
|
||
with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
|
||
fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
|
||
reproduce on build 10754 at this shape.
|
||
- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
|
||
thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
|
||
`finish_reason: "length"` with EMPTY `content` and half a thought in
|
||
`reasoning_content`: thinking tokens count against the budget, so a
|
||
short-budget vision probe reads as "the model is blind". Images do reach the
|
||
model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
|
||
alone). Keep the ≥256 px probe rule from the earlier eras.
|
||
- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
|
||
`failed to parse reasoning content (response_format json_schema is currently
|
||
only supported with parse_reasoning=true), content will be stored in
|
||
'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
|
||
prompt with ``. It is **cosmetic** — the full chain of thought
|
||
is in `content`. The OpenAI `/v1/chat/completions` path parses it into
|
||
`reasoning_content` correctly, which is the path the harness and every
|
||
consumer uses. Do not diagnose this as a fault, and do not drop verbosity
|
||
below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
|
||
- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
|
||
`/completion` request generated 0–1 tokens with no error (the model emits an
|
||
end-of-turn marker immediately, and a text context has no role frame to keep
|
||
it talking). Chat traffic is unaffected; nothing in the harness posts raw
|
||
`/completion` at depth.
|
||
|
||
## Ops notes
|
||
|
||
- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
|
||
must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
|
||
`BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
|
||
script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
|
||
for this payload is an OOM that looks like a model problem.
|
||
- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
|
||
refusal is possible on a compaction or subagent request. The CRACK re-export
|
||
(`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
|
||
`/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
|
||
`5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
|
||
byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
|
||
histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
|
||
ledger does not move. If a refusal ever surfaces, that is the fix; there is
|
||
also an MTP-bearing CRACK re-export upstream
|
||
(`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
|
||
needs the drafter — untested here.
|
||
- **Compaction hub unchanged:** all three endpoints still route
|
||
`compactionModel` here; 200,704 per slot is the fleet's only window that fits
|
||
a worst-case compaction input, and `--parallel 2` keeps compaction off the
|
||
critical path of a live generation.
|
||
- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
|
||
`maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
|
||
1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
|
||
became **load-bearing** (the template raises on any level outside
|
||
`xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
|
||
request, naming Jinja rather than the model.
|
||
- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
|
||
and the master CUDA build all remain on disk
|
||
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
|
||
`~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
|
||
page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
|
||
- **Deployed 2026-10-17 19:44** (`deploy.sh dankclaw --service frank-zealot
|
||
--service homepage`): boot ledger 22,664 used / 1,461 free, engine fit
|
||
projected 21,439 and left 2,119 ≥ 1,872 MiB, KV 13,328.00 MiB at
|
||
`n_ctx_seq = 200,704`, `n_slots = 2`, projector loaded; live probes green
|
||
(`/v1/models` serves `frank-zealot`, exact `pong` + `reasoning_content`,
|
||
`tool_calls` `{"city":"Seattle"}`, 512 px PNG → **Red**). e2e gate 33/33.
|
||
- **Picker row is live state:** the OpenWebUI `model` row's
|
||
`meta.description` named the 9B, so it was re-keyed in the database too
|
||
(backup `webui.db.bak-20261017a`). The row was written WITHOUT restarting
|
||
`open-webui` — the cached list refreshes on its next restart, and forcing one
|
||
would have logged every family session out for a cosmetic string. See
|
||
[[OpenWebUI]] for the picker trio.
|
||
|
||
## Related
|
||
|
||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
|
||
- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
|
||
- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
|
||
- [[deployment]] — deploy flow and incidents
|