Frank
695a5f3281
The cutover deploy's FIRST boot was not the shipped payload: a stale frank-zealot.service.d/ptq1.conf from the bench window still pointed FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health answered 200, /v1/models echoed the alias and the e2e gate passed 33/33. Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB, settled 17,922/6,203). Corrections the production ledger forced: - RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot. The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV. - Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over three streams (102.8 each). Wall-clock timing of a generation that stops on its own reads 4x low (72 t/s for a 24-token countdown) - measure with ignore_eos and read the engine's timings. - OpenWebUI stores both decoration values inside JSON columns (params/meta), and frank-dragoon's context.length had drifted to 262,144 against its own 200,704 ceiling; both re-keyed in the live DB. New law in AGENTS.md: after deploying a recently-benched LLM service, check DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot see the payload.
142 lines
8.9 KiB
Bash
142 lines
8.9 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
|
||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
|
||
# on the RTX 3090, :7771.
|
||
#
|
||
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export
|
||
# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third
|
||
# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what
|
||
# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships
|
||
# OFF — measured, not assumed; see the MTP law below, which is the whole story
|
||
# and the reason the publisher's "depth 2 works best" line does not transfer.
|
||
#
|
||
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
|
||
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
|
||
# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
|
||
# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
|
||
# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
|
||
# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
|
||
# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
|
||
# runtime for the 27B payload.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
|
||
# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
|
||
# head ships INSIDE the checkpoint, so MTP availability is a property of the
|
||
# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
|
||
# value; loading this file instead of that one is the only change that matters.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
|
||
# the publisher's card says the BF16 mmproj "can crash llama-server when several
|
||
# image requests run at once; Q8_0 is not affected", and this endpoint runs
|
||
# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER
|
||
# the engine's own fit check, so it is never inside a projection number.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
|
||
# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
|
||
# matched, NOT to this 9B file — if a refusal ever surfaces there is no
|
||
# one-line swap here; the 9B dealigned export would have to be fetched first.
|
||
#
|
||
# Served model id — the harness/family name. The SERVICE, unit, scripts and
|
||
# this env file keep the frank-zealot name; only the alias on the wire is the
|
||
# model id. Consumers that key on the served id: OpenWebUI (model row,
|
||
# access_grant, api_configs model_ids) and the harness models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per
|
||
# conversation — each slot at the model's TRAINED window, which is the point of
|
||
# the swap (the 27B era had to cut each conversation to 200,704 to fit two).
|
||
# MUST stay divisible by PARALLEL (the seam enforces it) and the harness
|
||
# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a
|
||
# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the
|
||
# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot
|
||
# would push each below the compaction contract.
|
||
CTX_SIZE=786432
|
||
PARALLEL=3
|
||
BATCH_SIZE=4096
|
||
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
|
||
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
|
||
# second job here: it sets the compute-buffer size, and the compute buffers are
|
||
# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
|
||
UBATCH_SIZE=2048
|
||
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
|
||
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
|
||
# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B
|
||
# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB —
|
||
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
|
||
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
|
||
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
|
||
# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
|
||
# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
|
||
# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
|
||
# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
|
||
# alone under-prices the shape.
|
||
CACHE_TYPE_K=q8_0
|
||
CACHE_TYPE_V=q8_0
|
||
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
|
||
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
|
||
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
|
||
# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers,
|
||
# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18):
|
||
# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
|
||
# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
|
||
# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
|
||
# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
|
||
# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
|
||
# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot
|
||
# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up —
|
||
# the squeeze, not the drafter, is what halves decode. The arm that WOULD run
|
||
# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5,
|
||
# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a
|
||
# card whose projector worst-case lands after the fit check. This endpoint's
|
||
# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3
|
||
# concurrent subagents, so the deployed arm takes the +17 % prefill
|
||
# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate
|
||
# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free.
|
||
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
|
||
# flags move as a set or the arm measures a crash instead of an absence.
|
||
SPEC_TYPE=none
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
|
||
# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
|
||
# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
|
||
# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
|
||
# PRESENCE_PENALTY="" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
|
||
# 17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
|
||
# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
|
||
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
|
||
# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
|
||
# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
|
||
# mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
|
||
# meta) — charged AFTER the engine's own fit check, as always
|
||
# no clamp line, no cudaMalloc error
|
||
# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
|
||
# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
|
||
# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
|
||
# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
|
||
# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
|
||
# warmup artifact (37.7 t/s class) and a natural generation that stops early
|
||
# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
|
||
# the engine's own timings.
|
||
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
|
||
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
|
||
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
|
||
# (−1,306 MiB measured on this shape).
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|
||
|
||
# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
|
||
# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
|
||
# this service has held the critical LlmEndpointDown alert for as long as the
|
||
# metric has history (verified 2026-10-18: the oldest retained sample of
|
||
# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
|
||
# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
|
||
# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|