Frank
3eaba57925
Frank ordered the adapter; this adds it to the shape as one knob and one flag. - `LORA_FILE` / `LORA_SCALE` in the env, `--lora-scaled <file>:<scale>` as ONE argv token so a scale can never reach the engine without its adapter. The rollback follows the mmproj convention exactly: `LORA_FILE=` (explicitly empty) drops the flag entirely, no rebuild, and `LORA_SCALE=0` is the card's own "inert but present" value. - New fetcher `scripts/download-frank-zealot-uncensored-lora.sh`: `-f`, byte count, sha256, `--verify-only`, idempotent. The file is NOT LFS (a plain git object), so the pinned sha is the content hash, not an LFS oid. - The deployer gates on the adapter beside the pack and the projector, and its error names the ADAPTER's fetcher — sending an operator to the bonsai9 script for a 3 MB file is how a payload ships without it. - Seam: path+scale pinned as one token, the scale knob including 0, absence on the empty opt-out, a refusal to boot on a missing adapter, and the env ↔ deployer anti-drift pair extended to LORA_FILE/LLM_ZEALOT_LORA. 454/454. Provenance, stated where it will be read: the adapter's own metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (3561df7a…) as the pack the direction was fitted against. This endpoint serves the PQ2_0+MTP+Q8_0 packing (c79fcdfa…), which is not that file and is not on this host. The loader matches tensor names, not hashes, so it applies — and build 10777 is the build the card verified on — but the strength is calibrated on a sibling quantization. The proof is therefore the refusal probe against the live endpoint, captured before and after, and not the card's table.
244 lines
13 KiB
Bash
Executable File
244 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||
# vendor build 10777, CUDA sm_86).
|
||
#
|
||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||
# an UNQUANTIZED f16 KV cache.
|
||
#
|
||
# 2026-10-19 (Frank): THREE 200,704-token conversations over a 602,112 pool
|
||
# (--ctx-size 602112 --parallel 3), K left at the engine's own default (f16),
|
||
# V dropped to q8_0, batch 1024×6 / ubatch 1024, and the MTP drafter ARMED
|
||
# (--spec-type draft-mtp). The 9B bills 8,192 elements per token per cache (8
|
||
# full-attention layers instead of the 27B's 16), so f16 K is 16,384 B/token and
|
||
# q8_0 V is 8,704: 25,088 B/token = 14,406.00 MiB at the 602,112 pool. That
|
||
# 7,680 B/token against exact V is exactly what funds the third conversation
|
||
# AND a draft context that bills f16 KV over the whole pool. Verbosity 4 ships
|
||
# so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in
|
||
# the journal.
|
||
#
|
||
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
||
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
||
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
||
# full layers (14,406.00 MiB of f16 K + q8_0 V at the 602,112 pool); the
|
||
# DeltaNet layers cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so
|
||
# 150.75 MiB for three, and an armed depth-2 drafter triples that again. The
|
||
# runtime is a hard lock, not a preference: custom GGML type 142 plus
|
||
# prism.hadamard.* metadata, which only the vendor build registers — and the
|
||
# b10776 gate that moved the prebuilt 10754 → 10777 (MTP beside a quantized V)
|
||
# is LOAD-BEARING again on this shape, not moot as it was under exact KV.
|
||
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
||
# server on concurrent images, and this endpoint runs three slots), so --mmproj is
|
||
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
||
# opt-out.
|
||
#
|
||
# A refusal-direction adapter rides on top of the pack (Frank, 2026-10-19):
|
||
# AtomicChat's rank-1 LoRA, applied at load by --lora-scaled. It is an edit
|
||
# inside the compute graph, not a second model — B = -r (the refusal direction),
|
||
# A built from the base pack's folded input basis, every transformer layer, the
|
||
# embedding deliberately untouched because an embedding LoRA crashes this fork's
|
||
# graph. The card verified it on prism-b10777-e3de715 — the build pinned here —
|
||
# and says older forks refuse the pack outright (`unsupported
|
||
# prism.hadamard.version: 2`), which makes the runtime lock doubly load-bearing.
|
||
# Its own metadata names the PTQ1_0 packing as the pack the direction was fitted
|
||
# against, so what proves the effect is a refusal probe against the live
|
||
# endpoint, not the metadata and not the card's table. LORA_FILE="" drops the
|
||
# flag and the endpoint is the plain pack again.
|
||
#
|
||
# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
|
||
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
|
||
# what made it lose that evening was the bill, not the drafts — the draft context
|
||
# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
|
||
# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the
|
||
# card, and it triples the recurrent state per slot (452.25 MiB for three cells
|
||
# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays
|
||
# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch
|
||
# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census
|
||
# showed — that figure does not transfer between shapes. SPEC_TYPE=none is
|
||
# the rollback and strips the whole flag set.
|
||
#
|
||
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
||
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||
# repeat_penalty 1.0 — confirmed in the engine's own /props defaults) and
|
||
# llama.cpp reads it; --presence-penalty is the one recipe value with no
|
||
# metadata key, so this script passes it. A request may still override per call
|
||
# (the fleet harness sends the same values).
|
||
#
|
||
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||
set -euo pipefail
|
||
|
||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||
if [[ -f "$ENV_FILE" ]]; then
|
||
set -a
|
||
# shellcheck disable=SC1090
|
||
source "$ENV_FILE"
|
||
set +a
|
||
fi
|
||
|
||
# --- defaults (used when the env file does not set them) -------------------
|
||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
|
||
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
||
: "${ALIAS:=frank-zealot}"
|
||
: "${BIND_ADDR:=0.0.0.0}"
|
||
: "${LISTEN_PORT:=7771}"
|
||
# ctx-size is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. The harness contextWindow MUST quote the PER-SLOT number — a
|
||
# longer prompt is answered with a 400, never truncated. The pool is priced by
|
||
# the KV bill, not by the trained window: at 25,088 B/token (f16 K + q8_0 V) the
|
||
# 602,112 pool costs 14,406.00 MiB and the armed drafter adds its own f16 KV over
|
||
# the whole pool (2,352.00 MiB). Exact V at this pool (18,816 MiB) or the old
|
||
# 786,432 pool at any dtype cannot boot inside the 24,123 MiB the driver leaves
|
||
# addressable.
|
||
: "${CTX_SIZE:=602112}"
|
||
: "${PARALLEL:=3}"
|
||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
|
||
# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
|
||
# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2
|
||
# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed.
|
||
: "${BATCH_SIZE:=6144}"
|
||
: "${UBATCH_SIZE:=1024}"
|
||
# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is
|
||
# visible in argv) + q8_0 V. 8,192 elements per token per cache (8 full-attention
|
||
# layers × 4 KV heads × 256 dim) ⇒ 16,384 B/token for K and 8,704 for V (q8_0 is
|
||
# 1.0625 B/element — llama.cpp has no true fp8 KV dtype), 25,088 B/token and
|
||
# 14,406.00 MiB at the 602,112 pool. That is the halfway door between exact KV
|
||
# (32,768 B/token, which costs the third conversation and the drafter both) and
|
||
# the fully quantized pair (17,408 B/token, which costs retrieval fidelity in the
|
||
# keys); --flash-attn stays pinned because quantized V is illegal without it.
|
||
: "${CACHE_TYPE_K:=f16}"
|
||
: "${CACHE_TYPE_V:=q8_0}"
|
||
# Speculative decoding defaults ON: the head IS in the checkpoint
|
||
# (qwen35.nextn_predict_layers = 1 — MTP is a property of the FILE, so the plain
|
||
# PQ2_0 sibling could not run it at any flag value) and the draft context rides
|
||
# the MAIN model's weights, so there is no --spec-draft-model. Its bill is the
|
||
# target's whole pool at 4,096 B/position plus ~0.45 GiB of buffers at this
|
||
# ubatch and a tripled recurrent state; CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024
|
||
# above are what pay for it. The draft cache dtype IS a further knob
|
||
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
|
||
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||
: "${SPEC_TYPE:=draft-mtp}"
|
||
: "${SPEC_DRAFT_N_MAX:=2}"
|
||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||
: "${SPEC_DRAFT_P_MIN:=0}"
|
||
: "${SPEC_DRAFT_FILE:=}"
|
||
# No colon on purpose, same convention as MMPROJ_FILE: an explicitly empty
|
||
# PRESENCE_PENALTY is the documented opt-out and must NOT be replaced by the
|
||
# default (a colon treats null as unset and would silently re-add the flag).
|
||
: "${PRESENCE_PENALTY=1.0}"
|
||
: "${NGL:=99}"
|
||
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||
# the VRAM-fit forensics need — the exact-KV size and the draft-KV bill are
|
||
# only provable from this log. Levels above 4 flood the journal.
|
||
: "${VERBOSITY:=4}"
|
||
# /metrics needs the flag: llama.cpp returns 501 unless the server starts with
|
||
# --metrics, and Dankclaw's Prometheus scrapes every fleet endpoint's /metrics —
|
||
# a 501 reads as DOWN, which is how the critical LlmEndpointDown alert ended up
|
||
# firing for this service ever since the metric has history at all — the oldest
|
||
# retained sample of up{job="frank-zealot"} (2026-10-11) is already 0, so this
|
||
# target has never been scraped successfully, across every payload era. The
|
||
# counters cost nothing on the request path;
|
||
# METRICS=0 is the documented opt-out.
|
||
: "${METRICS:=1}"
|
||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
||
# Refusal adapter. No colon on purpose, the same convention as MMPROJ_FILE: an
|
||
# explicitly empty LORA_FILE is the documented opt-out (the plain pack again)
|
||
# and must NOT be replaced by the default path. The scale travels INSIDE the
|
||
# same argv token as the file, so LORA_SCALE only matters while LORA_FILE names
|
||
# something; 0 is the card's "switch it off without touching the file" value.
|
||
: "${LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf}"
|
||
: "${LORA_SCALE:=1}"
|
||
|
||
# Fail loudly rather than half-loading: a missing file is a download that
|
||
# has not finished, and Restart=always would turn it into a crash loop that
|
||
# reads like a runtime fault.
|
||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||
done
|
||
EXTRA_ARGS=()
|
||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||
fi
|
||
|
||
# The adapter reaches the engine as ONE token — --lora-scaled <file>:<scale> —
|
||
# so a scale can never arrive without its file. It is applied once, at load;
|
||
# nothing here hot-swaps it (--lora-init-without-apply plus POST /lora-adapters
|
||
# is that shape, and a process holding three conversations is not the place to
|
||
# re-parameterise weights mid-flight). A missing file is a fetch that never
|
||
# finished, so refuse rather than let Restart=always loop on it.
|
||
if [[ -n "$LORA_FILE" ]]; then
|
||
[[ -e "$LORA_FILE" ]] || { echo "frank-zealot: missing $LORA_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")
|
||
fi
|
||
|
||
# Prometheus scraping is opt-IN in llama.cpp: without --metrics the server
|
||
# answers 501 on /metrics, and the fleet scrape then reports the endpoint DOWN.
|
||
if [[ "$METRICS" = "1" ]]; then
|
||
EXTRA_ARGS+=(--metrics)
|
||
fi
|
||
|
||
# Speculative flags move as a SET: they reach the argv ONLY when a drafter is
|
||
# named, and all of them together — a no-spec baseline must not carry half a
|
||
# spec pair. n-max 2 / n-min 0 / p-min 0 is the publisher's recipe ("Draft depth
|
||
# 2 works best"); the MTP head is IN the checkpoint (blk.32.nextn.*,
|
||
# qwen35.nextn_predict_layers = 1), so there is no --spec-draft-model here and
|
||
# the draft context rides the MAIN model's weights.
|
||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE"
|
||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN"
|
||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN")
|
||
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||
fi
|
||
fi
|
||
|
||
# The card's sampler recipe lives partly in the checkpoint: general.sampling.*
|
||
# (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0) is read by
|
||
# llama.cpp itself, and --repeat-last-n already defaults to 64. Presence penalty
|
||
# is the ONE value with no GGUF metadata key — the card says so explicitly — so
|
||
# it is the one knob that must reach the argv or the recipe is silently
|
||
# incomplete. PRESENCE_PENALTY="" is the documented opt-out.
|
||
if [[ -n "$PRESENCE_PENALTY" ]]; then
|
||
EXTRA_ARGS+=(--presence-penalty "$PRESENCE_PENALTY")
|
||
fi
|
||
|
||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||
# open after boot; make sure it exists before the server needs it.
|
||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||
|
||
# Flash attention stays pinned on. Exact f16 KV runs fine with or without it,
|
||
# but the documented fallback rung (quantized V) is ILLEGAL without it, and the
|
||
# build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17) so that rung is a
|
||
# real path — dropping the flag here would leave the retreat route silently
|
||
# falling back to something the env file does not describe.
|
||
ARGS=(
|
||
--model "$MODEL_FILE"
|
||
--alias "$ALIAS"
|
||
--host "$BIND_ADDR"
|
||
--port "$LISTEN_PORT"
|
||
--ctx-size "$CTX_SIZE"
|
||
--parallel "$PARALLEL"
|
||
--batch-size "$BATCH_SIZE"
|
||
--ubatch-size "$UBATCH_SIZE"
|
||
--n-gpu-layers "$NGL"
|
||
--flash-attn on
|
||
--cache-type-k "$CACHE_TYPE_K"
|
||
--cache-type-v "$CACHE_TYPE_V"
|
||
--log-verbosity "$VERBOSITY"
|
||
--jinja
|
||
--reasoning auto
|
||
--reasoning-format deepseek
|
||
)
|
||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|