# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
# on the RTX 3090, :7771.
#
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export,
# and the same evening the shape moved to TWO 262,144-token conversations over an
# UNQUANTIZED (f16) KV cache — "shift frank-zealot to 2 parallels of unquantized
# kv". Exact K and V is the retrieval-fidelity ask; the price is one concurrent
# conversation, because f16 owns the card. Speculative decoding ships OFF —
# measured, not assumed; see the MTP law below, which is the whole story and the
# reason the publisher's "depth 2 works best" line does not transfer (and why
# exact KV makes the drafter illegal outright).
#
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
# runtime for the 27B payload.
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
# head ships INSIDE the checkpoint, so MTP availability is a property of the
# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
# value; loading this file instead of that one is the only change that matters.
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
# the publisher's card says the BF16 mmproj "can crash llama-server when several
# image requests run at once; Q8_0 is not affected", and this endpoint runs
# --parallel 2, i.e. exactly that shape. The projector's worst case lands AFTER
# the engine's own fit check, so it is never inside a projection number.
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
# matched, NOT to this 9B file — if a refusal ever surfaces there is no
# one-line swap here; the 9B dealigned export would have to be fetched first.
#
# Served model id — the harness/family name. The SERVICE, unit, scripts and
# this env file keep the frank-zealot name; only the alias on the wire is the
# model id. Consumers that key on the served id: OpenWebUI (model row,
# access_grant, api_configs model_ids) and the harness models.yml.
ALIAS=frank-zealot
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 per
# conversation — each slot STILL at the model's TRAINED window; the per-slot
# contract did not move. MUST stay divisible by PARALLEL (the seam enforces it)
# and the harness contextWindow quotes the PER-SLOT number — a longer prompt is
# answered with a 400, never truncated. THE POOL CANNOT STAY AT 786,432 UNDER
# EXACT KV: f16 bills 32,768 B/token, so that pool is 24,576 MiB — more than the
# 24,123 MiB the driver leaves addressable, before a single weight loads.
# Halving the POOL to 524,288 (16,384 MiB) is what funds exact KV. The window
# cannot go past 262,144 a slot either: that is the native wall (no YaRN in this
# path), so a 3rd slot would be a KV-dtype cut, not more context.
CTX_SIZE=524288
PARALLEL=2
BATCH_SIZE=4096
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
# second job here: it sets the compute-buffer size, and the compute buffers are
# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
UBATCH_SIZE=2048
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
# 4 KV heads × 256 dim (n_embd_k_gqa = n_embd_v_gqa = 1,024 per layer) = 8,192
# elements per token per cache. EXACT KV: f16 = 2 B/element ⇒ 16,384 B per cache
# ⇒ 32,768 B/token = 32 KiB, so the 524,288 pool is 16,384.00 MiB. q8_0 bills
# 1.0625 B/element (32 codes + one 2-byte scale per 32-block) ⇒ 17,408 B/token,
# which is the 13,056.00 MiB the 3-slot ledger printed at the 786,432 pool.
# THE RATIO IS 1.882×, NOT 2×: the old note here said "f16 would bill 26,112
# MiB" because it doubled the q8_0 figure with its scale overhead included;
# exact arithmetic says 24,576 MiB at that pool. The wall stands, only its
# height moves. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
# instead of per token — the 3-slot ledger printed "RS buffer size = 150.75 MiB
# (3 cells, 32 layers, 3 seqs)", ~50.25 MiB a cell so two slots bill 100.50 —
# while the SPEC-ARMED arms print 452.25 MiB, because MTP depth 2 bills three
# seqs per slot (9 cells). Pool arithmetic alone under-prices the shape.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
# pool at its default dtype. That dtype IS a knob (--spec-draft-type-k /
# --spec-draft-type-v, both default f16 — verified against this build's --help),
# but SPEC_TYPE=none never sends them. Under the
# 3-slot q8_0 shape that bill was 4,096 B/position × 786,432 = 3,072.00 MiB plus
# ~1,758 MiB of buffers. Booted arms, same card, same traffic (2026-10-18, the
# WHOLE table is q8_0 K + q8_0 V — it predates exact KV):
#   3 × 262,144  spec OFF  ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
#   3 × 262,144  spec ON   ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
#   3 × 262,144  spec ON   ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
#   2 × 262,144  spec ON   ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
#   1 × 262,144  spec ON   ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
# MTP is worth +28 % decode whenever the card has room. At the SHIPPED exact-KV
# shape an f16-draft arm has none: the draft KV alone is 4,096 × 524,288 =
# 2,048.00 MiB plus its buffers, on a trunk that already projects ~21,200 MiB of
# the 24,123 the driver leaves addressable. DOORS BACK TO THE DRAFTER: one
# window per conversation; KV back to q8_0 with UBATCH_SIZE=1024 (213.5 t/s);
# or the UNMEASURED arm --spec-draft-type-k q8_0 --spec-draft-type-v q8_0, which
# halves the draft bill to 1,088 MiB and is legal on this build (>= b10776).
# Nobody has booted that beside exact KV — measure before promising it.
# This endpoint's load is still the fleet's COMPACTION HUB (prefill-bound,
# ~170k in): prefill is weight-bandwidth bound and does not read the KV cache
# per generated token, which is why exact KV is affordable here at all; what it
# buys back is concurrency — TWO conversations, not three.
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
# flags move as a set or the arm measures a crash instead of an absence.
SPEC_TYPE=none
SPEC_DRAFT_N_MAX=2
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0
# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
# PRESENCE_PENALTY="" is the documented opt-out.
PRESENCE_PENALTY=1.0
NGL=99
# Boot ledger REFERENCE — the previous production boot (2026-10-18 13:19,
# 3 × 262,144, q8_0 K + q8_0 V, verbosity 4), i.e. the shape this file replaced:
#   17,956 MiB used / 6,169 MiB free of 24,123 addressable (nvidia-smi settled;
#   its total column reads 24,576 — the driver reserves the difference, and
#   older notes quoting "a 24,123 MiB card" mean this addressable figure)
#   CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
#   CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
#   CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
#   compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
#   mmproj 595 MiB weights + 248.10 MiB CUDA compute meta; estimated worst case
#   868.32 MiB — charged AFTER the engine's own fit check, as always
#   no clamp line, no cudaMalloc error
# Measured on THAT boot (/completion timings, greedy, ignore_eos, 512 tokens):
# 168.7 t/s decode (three runs 168.4-168.7), ~300 t/s aggregate across THREE
# concurrent streams (102.8 each), 2,877 t/s prefill at 106,418. The 27B it
# replaced gave 76.6 / 109.4 / 691.7 at 166k. These are the numbers the exact-KV
# arm is judged against: decode reads 1.882× the KV bytes per token, so a decode
# drop is EXPECTED and must be measured, not assumed, while prefill should hold.
# Every non-KV term of that ledger survives this reslice except RS (−50.25 MiB,
# one fewer cell) and one slot's worth of scheduler state, so the f16 arm
# PROJECTS 17,956 − 13,056 + 16,384 − 50 = 21,234 MiB used / 2,889 free, and
# ~2.0 GiB free through the projector's worst case. REPLACE THIS PROJECTION WITH
# THE MEASURED PAIR after the boot: read the new ledger AND settled nvidia-smi.
# Fallback rungs if that margin gets eaten (desktop, co-tenant), cheapest first:
# CACHE_TYPE_V=q8_0 (exact K, quantized V) frees 8,192 MiB at this pool;
# CTX_SIZE=401408 (two 200,704 conversations) puts exact KV at 12,544.00 MiB and
# buys ~6.7 GiB back; UBATCH_SIZE=1024 hands 1,306 MiB of compute buffers back.
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
# and graph capture land in the journal, which is what the VRAM-fit forensics
# read. Levels above 4 flood the journal.
VERBOSITY=4

# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
# this service has held the critical LlmEndpointDown alert for as long as the
# metric has history (verified 2026-10-18: the oldest retained sample of
# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
METRICS=1