Frank
acbb539b97
Rebuilt the fork from origin/prism at 024474448 with the same configure the 10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja), installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds: libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base still registers PQ2_0, so type 142 and the hadamard metadata still load. LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH ($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the binary otherwise dies on libllama-server-impl.so. The unit test that pinned the 10777 path now asserts the invariant its own comment described (LLAMA_BIN names a llama-prism-* build) instead of a directory one rebuild happens to install; the suite runs on a box with no /home/dank, so existence is not checkable there. Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
244 lines
13 KiB
Bash
Executable File
244 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||
# vendor build 12118 (0.7.0-dev, commit 024474448), CUDA sm_86).
|
||
#
|
||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||
# an UNQUANTIZED f16 KV cache.
|
||
#
|
||
# 2026-10-19 (Frank): THREE 200,704-token conversations over a 602,112 pool
|
||
# (--ctx-size 602112 --parallel 3), K left at the engine's own default (f16),
|
||
# V dropped to q8_0, batch 1024×6 / ubatch 1024, and the MTP drafter ARMED
|
||
# (--spec-type draft-mtp). The 9B bills 8,192 elements per token per cache (8
|
||
# full-attention layers instead of the 27B's 16), so f16 K is 16,384 B/token and
|
||
# q8_0 V is 8,704: 25,088 B/token = 14,406.00 MiB at the 602,112 pool. That
|
||
# 7,680 B/token against exact V is exactly what funds the third conversation
|
||
# AND a draft context that bills f16 KV over the whole pool. Verbosity 4 ships
|
||
# so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in
|
||
# the journal.
|
||
#
|
||
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
||
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
||
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
||
# full layers (14,406.00 MiB of f16 K + q8_0 V at the 602,112 pool); the
|
||
# DeltaNet layers cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so
|
||
# 150.75 MiB for three, and an armed depth-2 drafter triples that again. The
|
||
# runtime is a hard lock, not a preference: custom GGML type 142 plus
|
||
# prism.hadamard.* metadata, which only the vendor build registers — and the
|
||
# b10776 gate that moved the prebuilt 10754 → 10777 (MTP beside a quantized V)
|
||
# is LOAD-BEARING again on this shape, not moot as it was under exact KV.
|
||
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
||
# server on concurrent images, and this endpoint runs three slots), so --mmproj is
|
||
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
||
# opt-out.
|
||
#
|
||
# A refusal-direction adapter rides on top of the pack (Frank, 2026-10-19):
|
||
# AtomicChat's rank-1 LoRA, applied at load by --lora-scaled. It is an edit
|
||
# inside the compute graph, not a second model — B = -r (the refusal direction),
|
||
# A built from the base pack's folded input basis, every transformer layer, the
|
||
# embedding deliberately untouched because an embedding LoRA crashes this fork's
|
||
# graph. The card verified it on prism-b10777-e3de715 — the build pinned here —
|
||
# and says older forks refuse the pack outright (`unsupported
|
||
# prism.hadamard.version: 2`), which makes the runtime lock doubly load-bearing.
|
||
# Its own metadata names the PTQ1_0 packing as the pack the direction was fitted
|
||
# against, so what proves the effect is a refusal probe against the live
|
||
# endpoint, not the metadata and not the card's table. LORA_FILE="" drops the
|
||
# flag and the endpoint is the plain pack again.
|
||
#
|
||
# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
|
||
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
|
||
# what made it lose that evening was the bill, not the drafts — the draft context
|
||
# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
|
||
# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the
|
||
# card, and it triples the recurrent state per slot (452.25 MiB for three cells
|
||
# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays
|
||
# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch
|
||
# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census
|
||
# showed — that figure does not transfer between shapes. SPEC_TYPE=none is
|
||
# the rollback and strips the whole flag set.
|
||
#
|
||
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
||
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||
# repeat_penalty 1.0 — confirmed in the engine's own /props defaults) and
|
||
# llama.cpp reads it; --presence-penalty is the one recipe value with no
|
||
# metadata key, so this script passes it. A request may still override per call
|
||
# (the fleet harness sends the same values).
|
||
#
|
||
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||
set -euo pipefail
|
||
|
||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||
if [[ -f "$ENV_FILE" ]]; then
|
||
set -a
|
||
# shellcheck disable=SC1090
|
||
source "$ENV_FILE"
|
||
set +a
|
||
fi
|
||
|
||
# --- defaults (used when the env file does not set them) -------------------
|
||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server}"
|
||
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
||
: "${ALIAS:=frank-zealot}"
|
||
: "${BIND_ADDR:=0.0.0.0}"
|
||
: "${LISTEN_PORT:=7771}"
|
||
# ctx-size is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. The harness contextWindow MUST quote the PER-SLOT number — a
|
||
# longer prompt is answered with a 400, never truncated. The pool is priced by
|
||
# the KV bill, not by the trained window: at 25,088 B/token (f16 K + q8_0 V) the
|
||
# 602,112 pool costs 14,406.00 MiB and the armed drafter adds its own f16 KV over
|
||
# the whole pool (2,352.00 MiB). Exact V at this pool (18,816 MiB) or the old
|
||
# 786,432 pool at any dtype cannot boot inside the 24,123 MiB the driver leaves
|
||
# addressable.
|
||
: "${CTX_SIZE:=602112}"
|
||
: "${PARALLEL:=3}"
|
||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
|
||
# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
|
||
# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2
|
||
# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed.
|
||
: "${BATCH_SIZE:=6144}"
|
||
: "${UBATCH_SIZE:=1024}"
|
||
# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is
|
||
# visible in argv) + q8_0 V. 8,192 elements per token per cache (8 full-attention
|
||
# layers × 4 KV heads × 256 dim) ⇒ 16,384 B/token for K and 8,704 for V (q8_0 is
|
||
# 1.0625 B/element — llama.cpp has no true fp8 KV dtype), 25,088 B/token and
|
||
# 14,406.00 MiB at the 602,112 pool. That is the halfway door between exact KV
|
||
# (32,768 B/token, which costs the third conversation and the drafter both) and
|
||
# the fully quantized pair (17,408 B/token, which costs retrieval fidelity in the
|
||
# keys); --flash-attn stays pinned because quantized V is illegal without it.
|
||
: "${CACHE_TYPE_K:=f16}"
|
||
: "${CACHE_TYPE_V:=q8_0}"
|
||
# Speculative decoding defaults ON: the head IS in the checkpoint
|
||
# (qwen35.nextn_predict_layers = 1 — MTP is a property of the FILE, so the plain
|
||
# PQ2_0 sibling could not run it at any flag value) and the draft context rides
|
||
# the MAIN model's weights, so there is no --spec-draft-model. Its bill is the
|
||
# target's whole pool at 4,096 B/position plus ~0.45 GiB of buffers at this
|
||
# ubatch and a tripled recurrent state; CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024
|
||
# above are what pay for it. The draft cache dtype IS a further knob
|
||
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||
# MTP needs fork build >= prism-b10776; LLAMA_BIN is 12118 (b10777 kept as rollback)
|
||
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||
: "${SPEC_TYPE:=draft-mtp}"
|
||
: "${SPEC_DRAFT_N_MAX:=2}"
|
||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||
: "${SPEC_DRAFT_P_MIN:=0}"
|
||
: "${SPEC_DRAFT_FILE:=}"
|
||
# No colon on purpose, same convention as MMPROJ_FILE: an explicitly empty
|
||
# PRESENCE_PENALTY is the documented opt-out and must NOT be replaced by the
|
||
# default (a colon treats null as unset and would silently re-add the flag).
|
||
: "${PRESENCE_PENALTY=1.0}"
|
||
: "${NGL:=99}"
|
||
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||
# the VRAM-fit forensics need — the exact-KV size and the draft-KV bill are
|
||
# only provable from this log. Levels above 4 flood the journal.
|
||
: "${VERBOSITY:=4}"
|
||
# /metrics needs the flag: llama.cpp returns 501 unless the server starts with
|
||
# --metrics, and Dankclaw's Prometheus scrapes every fleet endpoint's /metrics —
|
||
# a 501 reads as DOWN, which is how the critical LlmEndpointDown alert ended up
|
||
# firing for this service ever since the metric has history at all — the oldest
|
||
# retained sample of up{job="frank-zealot"} (2026-10-11) is already 0, so this
|
||
# target has never been scraped successfully, across every payload era. The
|
||
# counters cost nothing on the request path;
|
||
# METRICS=0 is the documented opt-out.
|
||
: "${METRICS:=1}"
|
||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
||
# Refusal adapter. No colon on purpose, the same convention as MMPROJ_FILE: an
|
||
# explicitly empty LORA_FILE is the documented opt-out (the plain pack again)
|
||
# and must NOT be replaced by the default path. The scale travels INSIDE the
|
||
# same argv token as the file, so LORA_SCALE only matters while LORA_FILE names
|
||
# something; 0 is the card's "switch it off without touching the file" value.
|
||
: "${LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf}"
|
||
: "${LORA_SCALE:=1}"
|
||
|
||
# Fail loudly rather than half-loading: a missing file is a download that
|
||
# has not finished, and Restart=always would turn it into a crash loop that
|
||
# reads like a runtime fault.
|
||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||
done
|
||
EXTRA_ARGS=()
|
||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||
fi
|
||
|
||
# The adapter reaches the engine as ONE token — --lora-scaled <file>:<scale> —
|
||
# so a scale can never arrive without its file. It is applied once, at load;
|
||
# nothing here hot-swaps it (--lora-init-without-apply plus POST /lora-adapters
|
||
# is that shape, and a process holding three conversations is not the place to
|
||
# re-parameterise weights mid-flight). A missing file is a fetch that never
|
||
# finished, so refuse rather than let Restart=always loop on it.
|
||
if [[ -n "$LORA_FILE" ]]; then
|
||
[[ -e "$LORA_FILE" ]] || { echo "frank-zealot: missing $LORA_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")
|
||
fi
|
||
|
||
# Prometheus scraping is opt-IN in llama.cpp: without --metrics the server
|
||
# answers 501 on /metrics, and the fleet scrape then reports the endpoint DOWN.
|
||
if [[ "$METRICS" = "1" ]]; then
|
||
EXTRA_ARGS+=(--metrics)
|
||
fi
|
||
|
||
# Speculative flags move as a SET: they reach the argv ONLY when a drafter is
|
||
# named, and all of them together — a no-spec baseline must not carry half a
|
||
# spec pair. n-max 2 / n-min 0 / p-min 0 is the publisher's recipe ("Draft depth
|
||
# 2 works best"); the MTP head is IN the checkpoint (blk.32.nextn.*,
|
||
# qwen35.nextn_predict_layers = 1), so there is no --spec-draft-model here and
|
||
# the draft context rides the MAIN model's weights.
|
||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE"
|
||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN"
|
||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN")
|
||
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||
fi
|
||
fi
|
||
|
||
# The card's sampler recipe lives partly in the checkpoint: general.sampling.*
|
||
# (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0) is read by
|
||
# llama.cpp itself, and --repeat-last-n already defaults to 64. Presence penalty
|
||
# is the ONE value with no GGUF metadata key — the card says so explicitly — so
|
||
# it is the one knob that must reach the argv or the recipe is silently
|
||
# incomplete. PRESENCE_PENALTY="" is the documented opt-out.
|
||
if [[ -n "$PRESENCE_PENALTY" ]]; then
|
||
EXTRA_ARGS+=(--presence-penalty "$PRESENCE_PENALTY")
|
||
fi
|
||
|
||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||
# open after boot; make sure it exists before the server needs it.
|
||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||
|
||
# Flash attention stays pinned on. Exact f16 KV runs fine with or without it,
|
||
# but the documented fallback rung (quantized V) is ILLEGAL without it, and the
|
||
# build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17) so that rung is a
|
||
# real path — dropping the flag here would leave the retreat route silently
|
||
# falling back to something the env file does not describe.
|
||
ARGS=(
|
||
--model "$MODEL_FILE"
|
||
--alias "$ALIAS"
|
||
--host "$BIND_ADDR"
|
||
--port "$LISTEN_PORT"
|
||
--ctx-size "$CTX_SIZE"
|
||
--parallel "$PARALLEL"
|
||
--batch-size "$BATCH_SIZE"
|
||
--ubatch-size "$UBATCH_SIZE"
|
||
--n-gpu-layers "$NGL"
|
||
--flash-attn on
|
||
--cache-type-k "$CACHE_TYPE_K"
|
||
--cache-type-v "$CACHE_TYPE_V"
|
||
--log-verbosity "$VERBOSITY"
|
||
--jinja
|
||
--reasoning auto
|
||
--reasoning-format deepseek
|
||
)
|
||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|