--- type: reference title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200704, f16 K + q8_0 V, MTP armed" description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws." tags: - llm - frank-zealot - dankclaw - llama-cpp - gguf - rtx3090 - endpoint timestamp: 2026-10-19 --- # frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP + refusal adapter, 3 × 200,704, f16 K + q8_0 V, MTP armed Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub** (three conversations at 200,704 each, the window the other two endpoints also cap at). One page endpoint: eras are rows below, not pages. Siblings: [[frank-archon]], [[frank-dragoon]]. Deploy with `./scripts/deploy.sh dankclaw --service frank-zealot`; swap mechanics in [[Model Swap and Rollback]], sizing math in [[Llama Cpp Serving Sizing]]. `scripts/frank-zealot.env` and `scripts/serve-frank-zealot.sh` are the authority for the shape; where any prose disagrees with them, the prose is the bug. ## Current shape | Item | Value | |---|---| | Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, LFS sha256 `c79fcdfaf6b7bf47…` | | Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) | | Adapter | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` — **3,155,424 B**, sha256 `861b9c099fb5cd81…` — a rank-1 refusal-direction LoRA (`adapter.type lora`, 128 tensors, every transformer layer, embedding untouched), applied at load by `--lora-scaled :1`. Its metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (`3561df7a…`) as the pack it was fitted against — **not this packing, and not on this host**; see Laws | | Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh [DEST]` (the adapter) — both `--verify-only`-able, `-f`, byte count and sha256 per file | | Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload | | Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` | | Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) | | Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · **`--lora-scaled Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf:1`** · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` | **Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block) `full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1` (the head **is in the file**), trained window 262,144, `general.sampling.*` 1.0/0.95/20. Only the 8 full layers bill per token and the DeltaNet layers add ~50 MiB of recurrent state **per slot**, so the shape is a single budget equation at 25,088 B/token (f16 K 16,384 + q8_0 V 8,704): **14,406.00 MiB** of cache over a 602,112 pool + **2,352.00 MiB** of draft cache (blk.32 at its own f16 rate, billed over the WHOLE pool) + ~7.7 GiB of weights, projector, buffers and states, inside the 24,123 MiB the driver leaves addressable. Exact V (32,768 B/token ⇒ 18,816 MiB here) buys neither the third conversation nor the drafter; the fully quantized pair (17,408 B/token) buys both and costs the keys their precision. **MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617 position, mean accepted length 2.41, and the publisher's card says depth 2 works best — and arming it is a budget decision, not a flag: the draft context is built against the *target's* pool, so `blk.32` bills 4,096 B/position (2,352.00 MiB at 602,112) plus 480.97 MiB of its own CUDA0 buffers, and it triples the recurrent state (452.25 MiB for three cells against 150.75 with the drafter off). `CACHE_TYPE_V=q8_0` is what pays that: 7,680 B/token cheaper than exact V = **4,410.00 MiB** at this pool. `UBATCH_SIZE=1024` — the rung every measured MTP arm here ran on — contributes only ~406 MiB at THIS shape, not the ~1,306 the 786,432-pool census showed, and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved decode (167.2 → 109.2 t/s). `--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's `--help`) would cut the draft's bill again and stay at their default; there is no `--spec-draft-model` — the draft rides the main model's weights, so a draft file would load the trunk twice. `SPEC_TYPE=none` is the rollback and strips the whole set. **The adapter is behaviour, not shape.** It folds 3.2 MB of rank-1 tensors into the trunk at load, so it moves nothing in the budget equation above — no KV, no slot, no window. What it changes is what the endpoint answers, and the only instrument for that is a refusal probe: the same two prompts the plain pack answered `I cannot provide…` (captured on the live endpoint before the swap, `temperature 0.1`), re-asked after it. Budget **≥ 2 048 tokens** for that probe on this payload — the thinking channel swallows a 600-token request whole and `content` comes back empty with `finish_reason: length`, which reads as a refusal that never happened. The adapter's own card table is **not** evidence here: it was measured on the PTQ1_0 packing. Rollback is `LORA_FILE=` (empty) in the env and one redeploy — no rebuild, no payload move; `LORA_SCALE=0` is the same result through the flag. **Measured on the cutover boot (2026-10-19, `02d04bc`).** Before: both probe prompts opened `I cannot provide…`. After: neither carries a refusal marker — one ran to its 2 048-token cap, the other finished at 1 930 — so the direction is out. It still volunteers a `⚠️ Important Disclaimer` heading before answering, which is style, not the circuit: the marker sentence is gone, and a leading hedge is not an escape hatch. Chat, tools (`tool_calls` with valid JSON) and vision ("Red", 281 prompt tokens) answered from the same boot, and `DropInPaths` came back empty. Read the ledger, not this paragraph, for provenance: the fork prints the adapter's `prism.abliterate.*` keys — including the `base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` line — at every start, so the packing mismatch is visible in the journal to anyone who boots it. **Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`): `contextWindow: 200704` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]` `compat.reasoningContentField: reasoning_content`, `compat.reasoningEffortMap` (**load-bearing**: the template takes only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the `compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here. **No rates, by law.** Frank ordered the reslice, not a number: the evidence a swap needs is the seam's fit gate, the deployer's health gate, and the boot ledger below. The previous era's exact-KV boot and its rate floors live in that era's row; those floors were measured while another consumer drove the other slot (56 prompts in three minutes, 96 % GPU-util), the exact condition a bench window exists to exclude. **The 2026-10-19 boot, measured** — verbosity-4 ledger of `ba69ea5`, started 03:21:56 PDT. Recorded here because projections drift, and nowhere else: | Term | Ledger | |---|---| | Target KV over the 602,112 pool | **14,406.00 MiB** — K (f16) 9,408.00 + V (q8_0) 4,998.00, `200704 cells, 8 layers, 3/3 seqs` | | Draft KV (`blk.32`, its own f16 context) | **2,352.00 MiB** — 1,176.00 + 1,176.00, `200704 cells, 1 layers, 3/3 seqs`: the whole pool, again | | Recurrent state | **452.25 MiB** — `3 cells, 32 layers, 3 seqs, 2 rs_seq`, R (f32) 20.25 + S (f32) 432.00 | | Compute buffers | target 1,665.00 CUDA0 + 424.88 CUDA_Host; draft context 480.97 + 424.86 | | Weights | 2,283.29 CUDA0 + 257.66 CPU-mapped | | Projector | 595.29 model + 248.10 compute + 24.93 CPU (its 868.32 worst case is still charged AFTER the engine's fit check) | | Engine's own verdict | projected 21,639 vs 23,404 free, then **"cannot meet free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"** | | Settled | **22,870 used / 1,255 free**, `n_slots = 3`, `n_ctx_slot = 200704`, `kv_unified = false` | This is the first boot on this card that does **not** meet llama.cpp's own free-memory target, and 1,255 MiB settled sits under the ~2 GiB band every previous era held. It boots, `/health` answers, and the seam's gate still passes — it priced 22,428 MiB, conservative by 442 against this ledger. The cheapest rungs back into the band are `CTX_SIZE=401408` (two conversations) or `SPEC_TYPE=none`; both are one line and neither changes the payload. **The drafter is engaged, measured on the same boot** (three ordinary requests, not a bench): acceptance 0.706 and 0.763 accepted-per-generated, mean accepted length 2.41 / 2.53, per-position rates (0.882, 0.765) and (0.871, 0.699) — depth 2 earning its keep at 0.3–1.5 MiB of draft state per saved prompt. The three probes that prove a swap: chat → `"zealot alive"` with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY content with `finish_reason: length` — the thinking-budget trap, not a fault); tools → `finish_reason: tool_calls` with `{"a":24,"b":7}`; vision → a 512 px solid-red PNG answers "Red" at `prompt_tokens` 280. **Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green: | Knob | Effect from the shipped shape | |---|---| | `SPEC_TYPE=none` | the drafter off: hands back its 2,352.00 MiB of cache, 480.97 MiB of CUDA0 buffers and the 301.50 MiB of rollback planes it adds to the recurrent state (~3.1 GiB), at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room | | `UBATCH_SIZE=2048` | — | | `CTX_SIZE=401408` | two 200,704 conversations: frees 4,802 MiB of cache — `contextWindow` MUST follow the per-slot number | | `CACHE_TYPE_V=f16` | exact V: +4,410 MiB at this pool with the drafter armed, which is a `cudaMalloc` death the seam refuses — legal only once the drafter or a slot is gone | | `CACHE_TYPE_K=q8_0` | the fully quantized pair (17,408 B/token): frees 4,410 MiB and moves the quantization error into the keys | ## Eras | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| | 2026-10-19 → | the same 9B MTP file and projector, **plus** `Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf` (3,155,424 B `861b9c09…`) applied by `--lora-scaled …:1` | PrismML fork 10777 (its verified build) | — (shape unchanged; the adapter is behaviour) | Frank ordered the refusal direction removed on this endpoint | one env line — `LORA_FILE=` (or `LORA_SCALE=0`) — and redeploy; the plain pack is the same file it already loads | | 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move | | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | | 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | — | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | | 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | — | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit | | 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` | | 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | — | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) | | 2026-10-03 → 10-05 | MiMo-V2.6-Distill-Qwen-9B Q6_K 7,793,710,624 B `ef96d05a…` + f16 mmproj 918,166,048 B | llama.cpp master CUDA | — | Frank ordered the uncensored 27B with MTP drafted | `/home/dank/models/MiMo-Q6K/` | | 2026-10-05 (committed `77decb5`→`380aafa`, **never deployed**) | Ornith-1.5-9B Q8_0 | llama.cpp master CUDA | 1 × 262,144, f16 K+V | Swift cut over the same evening while this deploy sat blocked at the whole-tree docs gate | `/home/dank/models/Ornith-Q8/`; revert to `380aafa`'s tree | | 2026-10-05 → 10-07 | Swift-1.5-27B-Uncensored RCO-IQ3_XXS-mtp 10,442,827,392 B `e13d3e74…` + publisher `mmproj-BF16.gguf` 931,145,888 B | llama.cpp master CUDA | — | the 9B Q6_K rung returned the 200k window and exact KV | `/home/dank/models/Swift15-UncIQ3XXS/`; the IQ3_S-mtp rung at `/home/dank/models/Swift15-IQ3S/`; commits `da36fa3` → `f2575a1` → `d8da7a9` | | 2026-10-02 → 10-03 | MiMo Q5_K_M 6,876,124,704 B `04cf46f2…` + the same f16 mmproj (hardlinked) | llama.cpp master CUDA | — | the ~3.5 GiB it left free bought the Q6_K rung | `/home/dank/models/MiMo-Q5KM/` | | 2026-10-01 → 10-02 | MiMo Q6_K (same file as above) | llama.cpp master CUDA | 2 × 262,144, f16 K + q8_0 V | one night: the Q5 rung bought the 2–3 GiB headroom band back | `/home/dank/models/MiMo-Q6K/` (same dir) | | 2026-10-01 (hours) | MiMo-V2.6 INT8-W8A16-AutoRound, 8 shards 12,891,718,440 B | SGLang in venv `~/GitPackages/sglang-cu128` (torch cu130 + an in-venv CUDA 13.0 nvcc) | — | llama.cpp was the only runtime exposing asymmetric per-tensor KV | payload purged — re-fetch `old/download-frank-zealot-sglang-int8.sh`; the venv is still on disk | | ≤ 2026-09 — 6 earlier eras: payloads, shapes and every measured number destroyed 2026-10-19 as obsolete | — | — | — | — | — ## Laws learned here - **Addressable ≠ advertised.** The card is 24,576 MiB but leaves **24,123 MiB**: quote `nvidia-smi`'s `free` field, never `total − used` — the driver's reservation is the difference. - **Exact KV and a 786,432 pool cannot coexist here**: 32,768 B/token ⇒ 24,576 MiB before a single weight loads (the retired "26,112 MiB" note doubled q8_0's scale overhead); the 27B analogue died at **25,088 MiB** with `cudaMalloc failed: out of memory`. Halving the pool funds exact KV; the per-slot window stops at the trained 262,144 (no YaRN here), so a third conversation is a KV-dtype decision, never more context. The wall's own figures: `cudaMalloc` failed on a **25,088.00 MiB** KV request when the engine's fit projection asked for **33,198 MiB** against **23,558 MiB** free. It booted f16 at all because the arm env's invented `KV_CACHE_TYPE` was swallowed by the serve script's `:=f16` default — so a failed window became the measurement. - **Exact KV reads 1.882× the bytes per generated token, not 2×** (32,768 vs 17,408 B/token, 2026-10-18): a decode drop of that order is EXPECTED and must be measured, while prefill should hold — and this service can afford the dtype precisely because its load is the compaction hub's ~170k-token prefill, which is weight-bandwidth bound and never re-reads the cache per generated token. The dtype is paid in decode and concurrency. - Never `--fit` with `parallel > 1` — it maximizes the *total* pool and shrinks each slot: set `CTX_SIZE` explicitly and keep it divisible by `PARALLEL` (the seam enforces it). - **KV rate law.** `full_layers × 4 KV heads × 256 dim` = **8,192 elements per token per cache** on this arch (the engine prints `n_embd_k_gqa = n_embd_v_gqa = 1024`), ×2 B (f16), ×1.0625 (q8_0 — 32 codes plus one 2-byte scale per 32-block), ×0.625 (q4_0): **32,768 / 17,408 / 10,240 B/token** on this 8-full-layer 9B, and 34,816 for a 16-full-layer 27B at q8_0. Everything else bills separately: DeltaNet recurrent state per **slot** (150.75 MiB for 3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair twice, 6.125 B/elem at f16 K + q8_0 V. - **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember. - **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 + q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is 873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot projected **21,439 MiB** and settled at **22,664**. Only `tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because projections drift. The same trap one knob down: the ubatch 2048 → 1024 step handed back **~1,306 MiB** at the 786,432 / q8_0-pair arm and **406 MiB** at the 602,112 / f16-K + q8_0-V boot (target compute 2,496.13 → 2,089.88, with CUDA0 going UP 257 while CUDA_Host went down 663). Changing the dtype or the pool moves the compute buffers too, so a credit measured at one shape is not a budget line at the next. The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV **7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it. The first 3-slot deploy (3 × 196,608) booted healthy and left only **1,153 MiB** free — that is how the 2–3 GiB band got violated in practice, and why the seam prices non-KV at the measured 9 GiB and refuses that shape. - **Ubatch is a prefill knob.** Decode never moves with it and the logical `--batch-size` is neutral; prefill climbs monotonically to **2048** on this CUDA card (+4.6 % at 8.9k depth; 710 → 1,188 t/s), a prompt shorter than the ubatch cannot see the knob at all (the control row), and a verdict never crosses backends — the 2× R9700 split peaks at 512. Each rung costs ~0.6–0.75 GiB of compute buffers; 2048 is the ceiling for a 27B-on-3090 shape. - **A quant rung is paid in decode**: Q5_K_M → Q6_K is +875 MiB of weights and **−10 %** rate on this bandwidth-bound card, and the byte-bandwidth law predicts a rung's rate within 1 %. - **Speculation pays only with room on the card.** The 2026-10-18 arm census (all q8_0 K+V at the 786,432 pool): 3 × 262,144 spec-OFF ubatch 2048 → 17,876 / 6,249 free, 167.2 t/s; spec-ON ubatch 2048 → 22,706 / 1,419, 109.2 t/s; spec-ON ubatch 1024 → 21,400 / 2,725, 213.5 t/s; 2 × 262,144 spec-ON → 213.2; 1 × 262,144 spec-ON → 214.2. That is **+28 % decode whenever the card has room**, and the doors back are one window per conversation, or q8_0 KV with ubatch 1024. The draft's own bill at that shape was 3,072.00 MiB of KV plus ~1,758 MiB of buffers. - Draft KV is built against the target's **whole** pool (~2.4 GiB measured on the 27B Swift boot, not the ~0.4 GiB a proposal-depth estimate suggests). Spec flags move as a SET — `SPEC_TYPE=none` strips all of them and half a set has segfaulted at load. Two context-backed drafters cannot share a process, so `draft-mtp,ngram-mod` is the only legal pair, and on a qwen35 hybrid ngram-mod disables recurrent-state rollback so rejected drafts degrade silently. - Acceptance alone never justifies speculation: 2026-08-25 measured 26.8–35.2 t/s drafting against 53.7 no-spec. - A checkpoint declaring `*_num_hidden_layers` MTP while carrying zero `mtp.*`/`nextn.*` tensors boots a 4.24 GiB ghost drafter at `accept rate: 0.00` — read the safetensors index, not the config. - MTP is a property of the **file**: the plain PQ2_0 sibling could not draft at any flag value; only the `-MTP` export carries `blk.N.nextn.*`. - **The format is the runtime lock.** PQ2_0 is `general.file_type 141` over 202 tensors of custom GGML type **142** plus `prism.hadamard.*` metadata: stock llama.cpp refuses the type, and a loader ignoring the Hadamard transform would dequant **wrong, silently** (same lock class as ROCmFPX). Build ≥ b10776 is additionally the floor that pairs MTP with a quantized V cache. - Check arch support in the source tree: `grep -c` on the packed binary returns 0 for every arch, including the one it serves. - **Base alignment.** Bonsai-2 is the censored base lineage. `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry (a one-line `MODEL_FILE` swap there, identical tensor census); no dealigned 9B exists, so for this payload the refusal escape hatch is a fetch, not a revert — which is exactly what was taken on 2026-10-19: the rank-1 adapter in the Adapter row above, behind `LORA_FILE`. - **A stale bench drop-in hijacks the production boot and no health check sees it.** An `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/.env` in `frank-zealot.service.d/` booted a 27B PTQ1_0 payload while `/health` answered 200, `/v1/models` echoed the alias flag and the e2e gate passed 33/33 — none of them read the payload. - After any bench window: `systemctl show -p DropInPaths` → `tr '\0' ' ' -/`, and payload decisions land on this page the moment they are made, or an approved arm becomes indistinguishable from a rogue one a day later. - **An arm env must use the serve script's own variable names** (`CACHE_TYPE_K`/`CACHE_TYPE_V`, `BIND_ADDR`, `NGL`): an invented `KV_CACHE_TYPE` is swallowed by `: "${CACHE_TYPE_K:=f16}"` and the arm boots f16 — an OOM that reads like a model fault. - **Stop, shebang, GPU node.** llama.cpp rides out the default stop timeout (`TimeoutStopSec=180` is on the unit for that reason) and a hand-booted arm keeps answering 7771 behind a `failed` unit — kill it **by PID**, never `pkill -f llama-server`. - `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check. - `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change. - **An adapter's provenance is metadata, not a gate.** Its GGUF carries `prism.abliterate.base_gguf = Ternary-Bonsai-2-9B-PTQ1_0.gguf` and that pack's sha256; the endpoint applies the file to the PQ2_0+MTP packing anyway, because the loader matches **tensor names, not hashes**. So an adapter's effect is only ever proven behaviourally (a refusal probe against the live endpoint), never by the card, and an adapter whose rank landed on `token_embd` would be a crash rather than a degradation — the publisher leaves the embedding untouched for exactly that reason. - **The fork pin now has a third reason.** The payload's custom type 142, the b10776 floor that pairs MTP with a quantized V, and now this: older PrismML builds (e.g. `prism-b10709`) refuse the adapter outright with `unsupported prism.hadamard.version: 2`, so a runtime that can load the pack can still be unable to carry the adapter. ## Contract - `/health` → `{"status":"ok"}`; `/props` reports `n_ctx` and `presence_penalty`; `/slots` shows occupancy; a fourth concurrent conversation **waits** for a slot — the slot count is the concurrency contract. - `/metrics` answers 200 with 15 `llamacpp:*` families **only with `--metrics`**; without it llama.cpp answers **501**, which Prometheus reads as DOWN while `/health` stays 200. The scrape target is `host.containers.internal:7771/metrics` and the oldest retained sample of `up{job="frank-zealot"}` (2026-10-11) is already 0 — this target was never successfully scraped, across every payload era. `METRICS=0` is the documented opt-out and re-breaks it; the counters cost nothing on the request path. - Reasoning arrives in **`reasoning_content`** (`--reasoning-format deepseek`) and **counts against `max_tokens`**: a short budget yields empty `content` with `finish_reason: "length"`, which reads as a broken model, not a mid-thought one. The slot prefix cache is live (`cached_tokens` in `usage`). - A client that sends no sampler is **not** greedy — the engine applies the payload's own `general.sampling.*` (1.0 / 0.95 / 20). `presence_penalty` is the one recipe value with no GGUF metadata key, so it must reach argv; the serve script uses `${PRESENCE_PENALTY=1.0}`, deliberately **not** `:=`, because `""` is the documented opt-out. - Vision: only the OpenAI **`image_url`** content part reaches the model — the legacy top-level `images` field bumps `usage.prompt_tokens` while the model invents an answer. Probe with a **≥ 256 px** solid color (a postage stamp answers "video") and budget **≥ 512** tokens; a 512 px image bills ~280 prompt tokens. - **Q8_0 mmproj, not BF16**: the publisher warns the BF16 tower "can crash llama-server when several image requests run at once", which is exactly the `--parallel 3` shape. `MMPROJ_FILE=` (explicitly empty) is the text-only opt-out and returns ~1.4 GiB; a path at a missing file is a boot **refusal**, never silent blindness. - Tools: native via `--jinja`; `finish_reason: tool_calls` with valid JSON arguments. - Two non-faults: a `400` naming 200,704 is one conversation outgrowing its **slot**, not pool exhaustion; and the raw `/completion` path is a **dead instrument on this fork build** — a greedy `ignore_eos` run answers **500 "The model produced output that does not match the expected Content-only format"**, because build 10777 guards the raw path the moment reasoning markup leaks (older builds only logged `failed to parse reasoning content` and answered anyway). The honest per-request instrument here is **`/v1/chat/completions`**, whose non-stream response carries the same `timings` object (`prompt_n`, `prompt_per_second`, `predicted_n` `predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes, while levels above 4 flood the journal. - llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`). - The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill / 54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`). ## Files - `scripts/frank-zealot.env` — **the authority for the values above** and nothing else: as of 2026-10-18 it carries the knobs plus the one-line reason a reader editing that line cannot recover here. The arm table, the reference boot ledger and the KV derivation all live on this page now; the header used to hold them, and two of its rung figures were wrong (`CACHE_TYPE_V=q8_0` and `CTX_SIZE=401408` both free **3,840 MiB** at the 524,288 pool, not 8,192 and not ~6.7 GiB — the Fallback rungs table above is the corrected statement). - `scripts/serve-frank-zealot.sh` — the exec line and its rationale comments; honors `FRANK_ZEALOT_ENV_FILE` for arm boots; exits 1 naming a missing binary, GGUF, projector or adapter rather than half-loading (a restart policy would turn an unfinished download into a crash loop that reads like a runtime fault). - `scripts/frank-zealot.service` — root unit, relinked into `/etc/systemd/system/`. - `scripts/download-frank-zealot-bonsai9.sh` (active payload + projector) · `scripts/download-frank-zealot-uncensored-lora.sh` (the refusal adapter; its header is the provenance record) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` · `scripts/download-frank-zealot.sh` (master-era GGUFs) · `old/download-frank-zealot-sglang-int8.sh` (SGLang). - `tests/frank_zealot_unit.bats` — stub-argv seam plus the VRAM fit gate. - `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates including the adapter, MainPID-hardened health); - `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml` `shared/grafana/provisioning/dashboards/llm-fleet.json` — the scrape, the alert, the dashboard. - Live state, not repo files: the OpenWebUI picker row (back up `webui.db` before re-keying) and `~/.omp/agent/models.yml`.