Frank
68afc6dbaf
Frank: "shift frank-zealot to 2 parallels of unquantized kv". Shape: CTX_SIZE 786432 -> 524288, PARALLEL 3 -> 2, CACHE_TYPE_K/V q8_0 -> f16. The per-slot window is UNCHANGED at 262,144 (the native trained window, and models.yml contextWindow), so what the exact cache buys is fidelity and what it pays is one concurrent conversation. The POOL is the term that had to move: exact KV bills 32,768 B/token at this geometry (8 full-attention layers x 4 KV heads x 256 dim, 2 B, K and V), so the old 786,432 pool asks 24,576 MiB against the 24,123 MiB the driver leaves addressable. 524,288 positions = 16,384 MiB, projected ~21,200 used / ~2,900 free from the 13:19 boot ledger (only the KV pool and one recurrent-state cell move). Also corrects a repo-wide arithmetic error: "f16 would bill 26,112 MiB" doubled the q8_0 ledger figure INCLUDING its 1/32 scale overhead. q8_0 is 1.0625 B/element, f16 exactly 2.0, so the true ratio is 1.882x. The wall stands; the number was wrong. And it corrects a false claim about the drafter: this build DOES expose --spec-draft-type-k / --spec-draft-type-v (verified against llama-server --help), so the draft cache is quantizable -- the exact-KV-plus-MTP exclusion is a measurement gap, not a law, and the 1,088 MiB quantized-draft arm is recorded as unmeasured. Seam: tests/frank_zealot_unit.bats re-pins the shape contract (2 slots, 524,288 pool, 262,144 per slot) and adds a VRAM fit gate that computes dtype x pool + the measured non-KV ledger terms against 24,123 MiB with >=1 GiB of slack -- and proves the gate REFUSES exact KV at the old pool, so passing means something. 18/18 green.
161 lines
10 KiB
Bash
161 lines
10 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
|
||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
|
||
# on the RTX 3090, :7771.
|
||
#
|
||
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export,
|
||
# and the same evening the shape moved to TWO 262,144-token conversations over an
|
||
# UNQUANTIZED (f16) KV cache — "shift frank-zealot to 2 parallels of unquantized
|
||
# kv". Exact K and V is the retrieval-fidelity ask; the price is one concurrent
|
||
# conversation, because f16 owns the card. Speculative decoding ships OFF —
|
||
# measured, not assumed; see the MTP law below, which is the whole story and the
|
||
# reason the publisher's "depth 2 works best" line does not transfer (and why
|
||
# exact KV makes the drafter illegal outright).
|
||
#
|
||
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
|
||
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
|
||
# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
|
||
# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
|
||
# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
|
||
# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
|
||
# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
|
||
# runtime for the 27B payload.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
|
||
# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
|
||
# head ships INSIDE the checkpoint, so MTP availability is a property of the
|
||
# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
|
||
# value; loading this file instead of that one is the only change that matters.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
|
||
# the publisher's card says the BF16 mmproj "can crash llama-server when several
|
||
# image requests run at once; Q8_0 is not affected", and this endpoint runs
|
||
# --parallel 2, i.e. exactly that shape. The projector's worst case lands AFTER
|
||
# the engine's own fit check, so it is never inside a projection number.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
|
||
# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
|
||
# matched, NOT to this 9B file — if a refusal ever surfaces there is no
|
||
# one-line swap here; the 9B dealigned export would have to be fetched first.
|
||
#
|
||
# Served model id — the harness/family name. The SERVICE, unit, scripts and
|
||
# this env file keep the frank-zealot name; only the alias on the wire is the
|
||
# model id. Consumers that key on the served id: OpenWebUI (model row,
|
||
# access_grant, api_configs model_ids) and the harness models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 per
|
||
# conversation — each slot STILL at the model's TRAINED window; the per-slot
|
||
# contract did not move. MUST stay divisible by PARALLEL (the seam enforces it)
|
||
# and the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||
# answered with a 400, never truncated. THE POOL CANNOT STAY AT 786,432 UNDER
|
||
# EXACT KV: f16 bills 32,768 B/token, so that pool is 24,576 MiB — more than the
|
||
# 24,123 MiB the driver leaves addressable, before a single weight loads.
|
||
# Halving the POOL to 524,288 (16,384 MiB) is what funds exact KV. The window
|
||
# cannot go past 262,144 a slot either: that is the native wall (no YaRN in this
|
||
# path), so a 3rd slot would be a KV-dtype cut, not more context.
|
||
CTX_SIZE=524288
|
||
PARALLEL=2
|
||
BATCH_SIZE=4096
|
||
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
|
||
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
|
||
# second job here: it sets the compute-buffer size, and the compute buffers are
|
||
# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
|
||
UBATCH_SIZE=2048
|
||
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
|
||
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
|
||
# 4 KV heads × 256 dim (n_embd_k_gqa = n_embd_v_gqa = 1,024 per layer) = 8,192
|
||
# elements per token per cache. EXACT KV: f16 = 2 B/element ⇒ 16,384 B per cache
|
||
# ⇒ 32,768 B/token = 32 KiB, so the 524,288 pool is 16,384.00 MiB. q8_0 bills
|
||
# 1.0625 B/element (32 codes + one 2-byte scale per 32-block) ⇒ 17,408 B/token,
|
||
# which is the 13,056.00 MiB the 3-slot ledger printed at the 786,432 pool.
|
||
# THE RATIO IS 1.882×, NOT 2×: the old note here said "f16 would bill 26,112
|
||
# MiB" because it doubled the q8_0 figure with its scale overhead included;
|
||
# exact arithmetic says 24,576 MiB at that pool. The wall stands, only its
|
||
# height moves. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||
# instead of per token — the 3-slot ledger printed "RS buffer size = 150.75 MiB
|
||
# (3 cells, 32 layers, 3 seqs)", ~50.25 MiB a cell so two slots bill 100.50 —
|
||
# while the SPEC-ARMED arms print 452.25 MiB, because MTP depth 2 bills three
|
||
# seqs per slot (9 cells). Pool arithmetic alone under-prices the shape.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=f16
|
||
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
|
||
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
|
||
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
|
||
# pool at its default dtype. That dtype IS a knob (--spec-draft-type-k /
|
||
# --spec-draft-type-v, both default f16 — verified against this build's --help),
|
||
# but SPEC_TYPE=none never sends them. Under the
|
||
# 3-slot q8_0 shape that bill was 4,096 B/position × 786,432 = 3,072.00 MiB plus
|
||
# ~1,758 MiB of buffers. Booted arms, same card, same traffic (2026-10-18, the
|
||
# WHOLE table is q8_0 K + q8_0 V — it predates exact KV):
|
||
# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
|
||
# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
|
||
# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
|
||
# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
|
||
# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
|
||
# MTP is worth +28 % decode whenever the card has room. At the SHIPPED exact-KV
|
||
# shape an f16-draft arm has none: the draft KV alone is 4,096 × 524,288 =
|
||
# 2,048.00 MiB plus its buffers, on a trunk that already projects ~21,200 MiB of
|
||
# the 24,123 the driver leaves addressable. DOORS BACK TO THE DRAFTER: one
|
||
# window per conversation; KV back to q8_0 with UBATCH_SIZE=1024 (213.5 t/s);
|
||
# or the UNMEASURED arm --spec-draft-type-k q8_0 --spec-draft-type-v q8_0, which
|
||
# halves the draft bill to 1,088 MiB and is legal on this build (>= b10776).
|
||
# Nobody has booted that beside exact KV — measure before promising it.
|
||
# This endpoint's load is still the fleet's COMPACTION HUB (prefill-bound,
|
||
# ~170k in): prefill is weight-bandwidth bound and does not read the KV cache
|
||
# per generated token, which is why exact KV is affordable here at all; what it
|
||
# buys back is concurrency — TWO conversations, not three.
|
||
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
|
||
# flags move as a set or the arm measures a crash instead of an absence.
|
||
SPEC_TYPE=none
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
|
||
# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
|
||
# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
|
||
# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
|
||
# PRESENCE_PENALTY="" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# Boot ledger REFERENCE — the previous production boot (2026-10-18 13:19,
|
||
# 3 × 262,144, q8_0 K + q8_0 V, verbosity 4), i.e. the shape this file replaced:
|
||
# 17,956 MiB used / 6,169 MiB free of 24,123 addressable (nvidia-smi settled;
|
||
# its total column reads 24,576 — the driver reserves the difference, and
|
||
# older notes quoting "a 24,123 MiB card" mean this addressable figure)
|
||
# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
|
||
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
|
||
# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
|
||
# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
|
||
# mmproj 595 MiB weights + 248.10 MiB CUDA compute meta; estimated worst case
|
||
# 868.32 MiB — charged AFTER the engine's own fit check, as always
|
||
# no clamp line, no cudaMalloc error
|
||
# Measured on THAT boot (/completion timings, greedy, ignore_eos, 512 tokens):
|
||
# 168.7 t/s decode (three runs 168.4-168.7), ~300 t/s aggregate across THREE
|
||
# concurrent streams (102.8 each), 2,877 t/s prefill at 106,418. The 27B it
|
||
# replaced gave 76.6 / 109.4 / 691.7 at 166k. These are the numbers the exact-KV
|
||
# arm is judged against: decode reads 1.882× the KV bytes per token, so a decode
|
||
# drop is EXPECTED and must be measured, not assumed, while prefill should hold.
|
||
# Every non-KV term of that ledger survives this reslice except RS (−50.25 MiB,
|
||
# one fewer cell) and one slot's worth of scheduler state, so the f16 arm
|
||
# PROJECTS 17,956 − 13,056 + 16,384 − 50 = 21,234 MiB used / 2,889 free, and
|
||
# ~2.0 GiB free through the projector's worst case. REPLACE THIS PROJECTION WITH
|
||
# THE MEASURED PAIR after the boot: read the new ledger AND settled nvidia-smi.
|
||
# Fallback rungs if that margin gets eaten (desktop, co-tenant), cheapest first:
|
||
# CACHE_TYPE_V=q8_0 (exact K, quantized V) frees 8,192 MiB at this pool;
|
||
# CTX_SIZE=401408 (two 200,704 conversations) puts exact KV at 12,544.00 MiB and
|
||
# buys ~6.7 GiB back; UBATCH_SIZE=1024 hands 1,306 MiB of compute buffers back.
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|
||
|
||
# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
|
||
# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
|
||
# this service has held the critical LlmEndpointDown alert for as long as the
|
||
# metric has history (verified 2026-10-18: the oldest retained sample of
|
||
# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
|
||
# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
|
||
# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|