# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
# on the RTX 3090, :7771.
#
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export
# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third
# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what
# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships
# OFF — measured, not assumed; see the MTP law below, which is the whole story
# and the reason the publisher's "depth 2 works best" line does not transfer.
#
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
# runtime for the 27B payload.
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
# head ships INSIDE the checkpoint, so MTP availability is a property of the
# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
# value; loading this file instead of that one is the only change that matters.
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
# the publisher's card says the BF16 mmproj "can crash llama-server when several
# image requests run at once; Q8_0 is not affected", and this endpoint runs
# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER
# the engine's own fit check, so it is never inside a projection number.
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
# matched, NOT to this 9B file — if a refusal ever surfaces there is no
# one-line swap here; the 9B dealigned export would have to be fetched first.
#
# Served model id — the harness/family name. The SERVICE, unit, scripts and
# this env file keep the frank-zealot name; only the alias on the wire is the
# model id. Consumers that key on the served id: OpenWebUI (model row,
# access_grant, api_configs model_ids) and the harness models.yml.
ALIAS=frank-zealot
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per
# conversation — each slot at the model's TRAINED window, which is the point of
# the swap (the 27B era had to cut each conversation to 200,704 to fit two).
# MUST stay divisible by PARALLEL (the seam enforces it) and the harness
# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a
# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the
# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot
# would push each below the compaction contract.
CTX_SIZE=786432
PARALLEL=3
BATCH_SIZE=4096
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
# second job here: it sets the compute-buffer size, and the compute buffers are
# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
UBATCH_SIZE=2048
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B
# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB —
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
# alone under-prices the shape.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers,
# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18):
#   3 × 262,144  spec OFF  ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
#   3 × 262,144  spec ON   ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
#   3 × 262,144  spec ON   ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
#   2 × 262,144  spec ON   ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
#   1 × 262,144  spec ON   ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot
# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up —
# the squeeze, not the drafter, is what halves decode. The arm that WOULD run
# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5,
# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a
# card whose projector worst-case lands after the fit check. This endpoint's
# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3
# concurrent subagents, so the deployed arm takes the +17 % prefill
# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate
# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free.
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
# flags move as a set or the arm measures a crash instead of an absence.
SPEC_TYPE=none
SPEC_DRAFT_N_MAX=2
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0
# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
# PRESENCE_PENALTY="" is the documented opt-out.
PRESENCE_PENALTY=1.0
NGL=99
# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
#   17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
#   CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
#   CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
#   CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
#   compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
#   mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
#   meta) — charged AFTER the engine's own fit check, as always
#   no clamp line, no cudaMalloc error
# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
# warmup artifact (37.7 t/s class) and a natural generation that stops early
# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
# the engine's own timings.
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
# (−1,306 MiB measured on this shape).
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
# and graph capture land in the journal, which is what the VRAM-fit forensics
# read. Levels above 4 flood the journal.
VERBOSITY=4

# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
# this service has held the critical LlmEndpointDown alert for as long as the
# metric has history (verified 2026-10-18: the oldest retained sample of
# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
METRICS=1