Frank
466fd477d8
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
135 lines
8.5 KiB
Bash
135 lines
8.5 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
|
||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
|
||
# on the RTX 3090, :7771.
|
||
#
|
||
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export
|
||
# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third
|
||
# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what
|
||
# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships
|
||
# OFF — measured, not assumed; see the MTP law below, which is the whole story
|
||
# and the reason the publisher's "depth 2 works best" line does not transfer.
|
||
#
|
||
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
|
||
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
|
||
# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
|
||
# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
|
||
# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
|
||
# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
|
||
# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
|
||
# runtime for the 27B payload.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
|
||
# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
|
||
# head ships INSIDE the checkpoint, so MTP availability is a property of the
|
||
# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
|
||
# value; loading this file instead of that one is the only change that matters.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
|
||
# the publisher's card says the BF16 mmproj "can crash llama-server when several
|
||
# image requests run at once; Q8_0 is not affected", and this endpoint runs
|
||
# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER
|
||
# the engine's own fit check, so it is never inside a projection number.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
|
||
# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
|
||
# matched, NOT to this 9B file — if a refusal ever surfaces there is no
|
||
# one-line swap here; the 9B dealigned export would have to be fetched first.
|
||
#
|
||
# Served model id — the harness/family name. The SERVICE, unit, scripts and
|
||
# this env file keep the frank-zealot name; only the alias on the wire is the
|
||
# model id. Consumers that key on the served id: OpenWebUI (model row,
|
||
# access_grant, api_configs model_ids) and the harness models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per
|
||
# conversation — each slot at the model's TRAINED window, which is the point of
|
||
# the swap (the 27B era had to cut each conversation to 200,704 to fit two).
|
||
# MUST stay divisible by PARALLEL (the seam enforces it) and the harness
|
||
# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a
|
||
# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the
|
||
# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot
|
||
# would push each below the compaction contract.
|
||
CTX_SIZE=786432
|
||
PARALLEL=3
|
||
BATCH_SIZE=4096
|
||
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
|
||
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
|
||
# second job here: it sets the compute-buffer size, and the compute buffers are
|
||
# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
|
||
UBATCH_SIZE=2048
|
||
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
|
||
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
|
||
# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B
|
||
# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB —
|
||
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
|
||
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
|
||
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
|
||
# bill a fixed recurrent state per SLOT instead of per token — the ledger's
|
||
# "RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)" = 150.75 MiB a
|
||
# slot — which is why pool arithmetic alone under-prices the shape.
|
||
CACHE_TYPE_K=q8_0
|
||
CACHE_TYPE_V=q8_0
|
||
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
|
||
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
|
||
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
|
||
# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers,
|
||
# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18):
|
||
# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
|
||
# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
|
||
# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
|
||
# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
|
||
# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
|
||
# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot
|
||
# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up —
|
||
# the squeeze, not the drafter, is what halves decode. The arm that WOULD run
|
||
# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5,
|
||
# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a
|
||
# card whose projector worst-case lands after the fit check. This endpoint's
|
||
# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3
|
||
# concurrent subagents, so the deployed arm takes the +17 % prefill
|
||
# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate
|
||
# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free.
|
||
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
|
||
# flags move as a set or the arm measures a crash instead of an absence.
|
||
SPEC_TYPE=none
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
|
||
# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
|
||
# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
|
||
# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
|
||
# PRESENCE_PENALTY="" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# Boot ledger (2026-10-18, this exact shape, nvidia-smi settled):
|
||
# 17,876 MiB used / 6,249 MiB free of 24,123
|
||
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
|
||
# CUDA0 RS 452.25 MiB (3 recurrent-state cells)
|
||
# compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB
|
||
# no clamp line, no cudaMalloc error
|
||
# Measured on the same boot: 167.2 t/s greedy decode (/completion, 512 tokens),
|
||
# 158.9 t/s single-stream chat, 280.2 t/s aggregate across THREE concurrent
|
||
# streams (93.8 each), 2,877 t/s prefill at 106,418 tokens. The 27B record it
|
||
# replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at
|
||
# 166k — beaten on every axis. The FIRST request after a boot is a warmup
|
||
# artifact (37.7 t/s class); never report it.
|
||
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
|
||
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
|
||
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
|
||
# (−1,306 MiB measured on this shape).
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|
||
|
||
# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
|
||
# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
|
||
# this service has held the critical LlmEndpointDown alert for as long as the
|
||
# metric has history (verified 2026-10-18: the oldest retained sample of
|
||
# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
|
||
# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
|
||
# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|