Frank
466fd477d8
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
227 lines
15 KiB
Markdown
227 lines
15 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10754), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (RETIRED, rollback target)"
|
||
description: "The frank-zealot runbook of 2026-10-17 → 2026-10-18 (RETIRED — rollback target) — the PrismML fork build 10754 of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape. Superseded by the 9B MTP export at 3 × 262,144."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
timestamp: 2026-10-17
|
||
---
|
||
|
||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
|
||
|
||
**RETIRED 2026-10-17 → 2026-10-18** (AGENTS.md Runbook Index), superseded by
|
||
[[frank-zealot-llama-prism-bonsai9-mtp]] — prism-ml/Ternary-Bonsai-2-**9B**
|
||
PQ2_0 **with the MTP head**, THREE conversations at the native 262,144 window
|
||
on fork build 10777. This era stays the **rollback target**: the payload
|
||
(`/home/dank/models/Bonsai2-PQ2_0/`) and the 10754 fork tree
|
||
(`~/GitPackages/llama-prism/`) are both still on disk. It owns the laws below —
|
||
the 34,816 B/token KV figure over 16 full-attention layers, the 25,088 MiB f16
|
||
OOM wall, and the two-boot VRAM pricing that every later shape on this card
|
||
reused.
|
||
|
||
## 2026-10-18 bench window — production unit stopped, candidate arm on 7771
|
||
|
||
As of 2026-10-18 frank-zealot sits in a deliberate, dated **bench window**
|
||
(fleet Benchmarking Policy: stop the production unit, bench candidate boots
|
||
against the service itself). The journal shows the stop at 03:38:31 and the
|
||
known llama.cpp shutdown shape — SIGTERM ignored past `TimeoutStopSec`,
|
||
`State 'stop-sigterm' timed out. Killing.` at 03:40:01 — after which the unit
|
||
reports `failed` (`is-enabled` → enabled); self-healing is merely parked
|
||
while the window runs. What answers 7771 is a **hand-booted candidate arm**
|
||
(`llama-prism-b10777/llama-server` on the 9B MTP file
|
||
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, alias
|
||
`frank-zealot`), not this runbook's payload: candidate-arm flags are not a
|
||
state — the pid and shape moved between two censuses ten minutes apart (one
|
||
argv read `--ctx-size 786432 --parallel 3 --spec-type draft-mtp`; a later
|
||
`/props` read `n_ctx 262144`, `speculative.types=none`); quote neither, and
|
||
expect `/v1/models` to report the candidate until the window closes. The
|
||
window's remedy is the redeploy: `./scripts/deploy.sh dankclaw --service
|
||
frank-zealot`. The production trio below is unchanged — the last PRODUCTION
|
||
boot under the unit (03:14–03:38 the same morning) printed the contract
|
||
exactly: `n_ctx = 401408`, `n_ctx_seq = 200704`, `CUDA0 KV buffer size =
|
||
13328.00 MiB`, 2 slots (journal).
|
||
|
||
**Window CLOSED 2026-10-18:** that candidate became production. The redeploy
|
||
shipped the 9B MTP export at `3 × 262,144` on fork build 10777 — see
|
||
[[frank-zealot-llama-prism-bonsai9-mtp]]; this payload is the rollback target.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
|
||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
|
||
| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
|
||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||
| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
|
||
| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
|
||
| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
|
||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
|
||
|
||
## Why the fork — the format is the lock
|
||
|
||
Read from the file's own metadata, not the model card:
|
||
|
||
|Key|Value|
|
||
|---|---|
|
||
|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
|
||
|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
|
||
|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
|
||
|Widths|embedding 5,120, feed-forward 17,408|
|
||
|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
|
||
|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
|
||
|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
|
||
|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
|
||
|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
|
||
|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
|
||
|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
|
||
|
||
Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
|
||
stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
|
||
same read), and even a loader that parsed it would ignore the
|
||
`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
|
||
as the ROCmFPX custom-type era. The vendor build 10754 already carries the
|
||
September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
|
||
-capable CUDA backend — proven by q8_0 V serving under flash attention at the
|
||
correct 34,816 B/token, below).
|
||
|
||
## VRAM census (2026-10-17, four arms, production stopped)
|
||
|
||
The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
|
||
all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
|
||
mmproj attached:
|
||
|
||
|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
|
||
|---|---|---|---|---|
|
||
|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
|
||
|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
|
||
|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
|
||
|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
|
||
|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
|
||
|
||
**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
|
||
16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
|
||
so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
|
||
(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
|
||
the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
|
||
context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
|
||
arithmetic alone under-prices the shape by that much.
|
||
|
||
**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
|
||
usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
|
||
settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
|
||
CUDA context. This is the difference between "2.6 GiB free on paper" and the
|
||
real **1.4 GiB** — the same late-allocation failure class that killed the
|
||
DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
|
||
`nvidia-smi`, never the projection.
|
||
|
||
## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
|
||
|
||
|Probe|Result|
|
||
|---|---|
|
||
|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
|
||
|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
|
||
|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
|
||
|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
|
||
|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
|
||
|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
|
||
|
||
Production boot (the 19:44 cutover deploy, same binary/flags/file): **75.8 t/s**
|
||
single-stream decode at a short prompt — within 1 % of the trial arm, so the
|
||
ladder above carries to the shipped shape.
|
||
|
||
## Behavior contract (probes, not assumptions)
|
||
|
||
- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
|
||
- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
|
||
with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
|
||
fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
|
||
reproduce on build 10754 at this shape.
|
||
- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
|
||
thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
|
||
`finish_reason: "length"` with EMPTY `content` and half a thought in
|
||
`reasoning_content`: thinking tokens count against the budget, so a
|
||
short-budget vision probe reads as "the model is blind". Images do reach the
|
||
model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
|
||
alone). Keep the ≥256 px probe rule from the earlier eras.
|
||
- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
|
||
`failed to parse reasoning content (response_format json_schema is currently
|
||
only supported with parse_reasoning=true), content will be stored in
|
||
'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
|
||
prompt with ``. It is **cosmetic** — the full chain of thought
|
||
is in `content`. The OpenAI `/v1/chat/completions` path parses it into
|
||
`reasoning_content` correctly, which is the path the harness and every
|
||
consumer uses. Do not diagnose this as a fault, and do not drop verbosity
|
||
below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
|
||
- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
|
||
`/completion` request generated 0–1 tokens with no error (the model emits an
|
||
end-of-turn marker immediately, and a text context has no role frame to keep
|
||
it talking). Chat traffic is unaffected; nothing in the harness posts raw
|
||
`/completion` at depth.
|
||
|
||
## Ops notes
|
||
|
||
- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
|
||
must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
|
||
`BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
|
||
script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
|
||
for this payload is an OOM that looks like a model problem.
|
||
- **`/metrics` needs `--metrics` (fixed 2026-10-18):** the fork answers **501 Not
|
||
Implemented** without the flag, and Dankclaw's Prometheus scrapes `/metrics`, so
|
||
`up{job="frank-zealot"}` has been 0 for the whole retained metric history — the
|
||
oldest sample (2026-10-11) is already 0 — while the endpoint itself answered
|
||
`/health` 200 the entire time. `METRICS=1` in `scripts/frank-zealot.env` now
|
||
ships the flag and the seam asserts it reaches the argv. halogen serves
|
||
`/metrics` natively, which is why that target was always green.
|
||
- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
|
||
refusal is possible on a compaction or subagent request. The CRACK re-export
|
||
(`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
|
||
`/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
|
||
`5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
|
||
byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
|
||
histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
|
||
ledger does not move. If a refusal ever surfaces, that is the fix; there is
|
||
also an MTP-bearing CRACK re-export upstream
|
||
(`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
|
||
needs the drafter — untested here.
|
||
- **Compaction hub unchanged:** all three endpoints still route
|
||
`compactionModel` here; 200,704 per slot is the fleet's only window that fits
|
||
a worst-case compaction input, and `--parallel 2` keeps compaction off the
|
||
critical path of a live generation.
|
||
- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
|
||
`maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
|
||
1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
|
||
became **load-bearing** (the template raises on any level outside
|
||
`xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
|
||
request, naming Jinja rather than the model.
|
||
- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
|
||
and the master CUDA build all remain on disk
|
||
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
|
||
`~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
|
||
page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
|
||
- **Deployed 2026-10-17 19:44** (`deploy.sh dankclaw --service frank-zealot
|
||
--service homepage`): boot ledger 22,664 used / 1,461 free, engine fit
|
||
projected 21,439 and left 2,119 ≥ 1,872 MiB, KV 13,328.00 MiB at
|
||
`n_ctx_seq = 200,704`, `n_slots = 2`, projector loaded; live probes green
|
||
(`/v1/models` serves `frank-zealot`, exact `pong` + `reasoning_content`,
|
||
`tool_calls` `{"city":"Seattle"}`, 512 px PNG → **Red**). e2e gate 33/33.
|
||
- **Picker row is live state:** the OpenWebUI `model` row's
|
||
`meta.description` named the 9B, so it was re-keyed in the database too
|
||
(backup `webui.db.bak-20261017a`). The row was written WITHOUT restarting
|
||
`open-webui` — the cached list refreshes on its next restart, and forcing one
|
||
would have logged every family session out for a cosmetic string. See
|
||
[[OpenWebUI]] for the picker trio.
|
||
|
||
## Related
|
||
|
||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
|
||
- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
|
||
- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
|
||
- [[deployment]] — deploy flow and incidents
|