Frank
30e8ed55a0
The boot of record for the new shape, from the verbosity-4 journal plus
settled nvidia-smi: target KV 14,406.00 (K f16 9,408.00 + V q8_0 4,998.00),
draft KV 2,352.00 over the same 200,704 cells x 3 seqs, recurrent state
452.25, compute 1,665.00 + 424.88 (draft 480.97 + 424.86), weights 2,283.29
+ 257.66, projector 595.29 + 248.10 + 24.93, settled 22,870 used / 1,255
free with n_slots = 3 / n_ctx_slot = 200704.
Two inherited numbers did not survive it, and both are corrected in the env,
the serve rationale and the page:
- The ubatch 2048 -> 1024 step was quoted at "-~1,306 MiB". That was the
786,432 / q8_0-pair arm. At THIS shape the target's compute buffers went
2,496.13 -> 2,089.88: 406 MiB, with CUDA0 going UP 257 while CUDA_Host
went down 663. q8_0 V (4,410 MiB, not the 3,840 previously written) is
what funds the drafter, alone.
- The fit gate priced 22,428 against a real 22,870: conservative by 442
MiB, so it stays honest as written.
And one finding stated plainly: this is the first boot on this card that
misses llama.cpp's own free-memory target - the engine printed "cannot meet
free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"
and settled at 1,255 MiB free, under the ~2 GiB band every previous era
held. Cheapest rungs back are CTX_SIZE=401408 or SPEC_TYPE=none, both one
line.
Probes on the same boot go in too: chat turn (448 tokens, 1,755 chars of
thinking - empty content at max_tokens 64 is the documented trap), tools
(finish_reason tool_calls, {"a":24,"b":7}), vision (512 px solid red ->
"Red", 280 prompt tokens), and the drafter's own statistics (acceptance
0.706 / 0.763, mean accepted length 2.41 / 2.53).
213 lines
11 KiB
Bash
Executable File
213 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
|
||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||
# vendor build 10777, CUDA sm_86).
|
||
#
|
||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||
# an UNQUANTIZED f16 KV cache.
|
||
#
|
||
# 2026-10-19 (Frank): THREE 200,704-token conversations over a 602,112 pool
|
||
# (--ctx-size 602112 --parallel 3), K left at the engine's own default (f16),
|
||
# V dropped to q8_0, batch 1024×6 / ubatch 1024, and the MTP drafter ARMED
|
||
# (--spec-type draft-mtp). The 9B bills 8,192 elements per token per cache (8
|
||
# full-attention layers instead of the 27B's 16), so f16 K is 16,384 B/token and
|
||
# q8_0 V is 8,704: 25,088 B/token = 14,406.00 MiB at the 602,112 pool. That
|
||
# 7,680 B/token against exact V is exactly what funds the third conversation
|
||
# AND a draft context that bills f16 KV over the whole pool. Verbosity 4 ships
|
||
# so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in
|
||
# the journal.
|
||
#
|
||
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
||
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
||
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
||
# full layers (14,406.00 MiB of f16 K + q8_0 V at the 602,112 pool); the
|
||
# DeltaNet layers cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so
|
||
# 150.75 MiB for three, and an armed depth-2 drafter triples that again. The
|
||
# runtime is a hard lock, not a preference: custom GGML type 142 plus
|
||
# prism.hadamard.* metadata, which only the vendor build registers — and the
|
||
# b10776 gate that moved the prebuilt 10754 → 10777 (MTP beside a quantized V)
|
||
# is LOAD-BEARING again on this shape, not moot as it was under exact KV.
|
||
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
||
# server on concurrent images, and this endpoint runs three slots), so --mmproj is
|
||
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
||
# opt-out.
|
||
#
|
||
# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
|
||
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
|
||
# what made it lose that evening was the bill, not the drafts — the draft context
|
||
# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
|
||
# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the
|
||
# card, and it triples the recurrent state per slot (452.25 MiB for three cells
|
||
# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays
|
||
# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch
|
||
# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census
|
||
# showed — that figure does not transfer between shapes. SPEC_TYPE=none is
|
||
# the rollback and strips the whole flag set.
|
||
#
|
||
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
||
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||
# repeat_penalty 1.0 — confirmed in the engine's own /props defaults) and
|
||
# llama.cpp reads it; --presence-penalty is the one recipe value with no
|
||
# metadata key, so this script passes it. A request may still override per call
|
||
# (the fleet harness sends the same values).
|
||
#
|
||
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||
set -euo pipefail
|
||
|
||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||
if [[ -f "$ENV_FILE" ]]; then
|
||
set -a
|
||
# shellcheck disable=SC1090
|
||
source "$ENV_FILE"
|
||
set +a
|
||
fi
|
||
|
||
# --- defaults (used when the env file does not set them) -------------------
|
||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
|
||
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
|
||
: "${ALIAS:=frank-zealot}"
|
||
: "${BIND_ADDR:=0.0.0.0}"
|
||
: "${LISTEN_PORT:=7771}"
|
||
# ctx-size is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. The harness contextWindow MUST quote the PER-SLOT number — a
|
||
# longer prompt is answered with a 400, never truncated. The pool is priced by
|
||
# the KV bill, not by the trained window: at 25,088 B/token (f16 K + q8_0 V) the
|
||
# 602,112 pool costs 14,406.00 MiB and the armed drafter adds its own f16 KV over
|
||
# the whole pool (2,352.00 MiB). Exact V at this pool (18,816 MiB) or the old
|
||
# 786,432 pool at any dtype cannot boot inside the 24,123 MiB the driver leaves
|
||
# addressable.
|
||
: "${CTX_SIZE:=602112}"
|
||
: "${PARALLEL:=3}"
|
||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
|
||
# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
|
||
# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2
|
||
# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed.
|
||
: "${BATCH_SIZE:=6144}"
|
||
: "${UBATCH_SIZE:=1024}"
|
||
# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is
|
||
# visible in argv) + q8_0 V. 8,192 elements per token per cache (8 full-attention
|
||
# layers × 4 KV heads × 256 dim) ⇒ 16,384 B/token for K and 8,704 for V (q8_0 is
|
||
# 1.0625 B/element — llama.cpp has no true fp8 KV dtype), 25,088 B/token and
|
||
# 14,406.00 MiB at the 602,112 pool. That is the halfway door between exact KV
|
||
# (32,768 B/token, which costs the third conversation and the drafter both) and
|
||
# the fully quantized pair (17,408 B/token, which costs retrieval fidelity in the
|
||
# keys); --flash-attn stays pinned because quantized V is illegal without it.
|
||
: "${CACHE_TYPE_K:=f16}"
|
||
: "${CACHE_TYPE_V:=q8_0}"
|
||
# Speculative decoding defaults ON: the head IS in the checkpoint
|
||
# (qwen35.nextn_predict_layers = 1 — MTP is a property of the FILE, so the plain
|
||
# PQ2_0 sibling could not run it at any flag value) and the draft context rides
|
||
# the MAIN model's weights, so there is no --spec-draft-model. Its bill is the
|
||
# target's whole pool at 4,096 B/position plus ~0.45 GiB of buffers at this
|
||
# ubatch and a tripled recurrent state; CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024
|
||
# above are what pay for it. The draft cache dtype IS a further knob
|
||
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
|
||
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||
: "${SPEC_TYPE:=draft-mtp}"
|
||
: "${SPEC_DRAFT_N_MAX:=2}"
|
||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||
: "${SPEC_DRAFT_P_MIN:=0}"
|
||
: "${SPEC_DRAFT_FILE:=}"
|
||
# No colon on purpose, same convention as MMPROJ_FILE: an explicitly empty
|
||
# PRESENCE_PENALTY is the documented opt-out and must NOT be replaced by the
|
||
# default (a colon treats null as unset and would silently re-add the flag).
|
||
: "${PRESENCE_PENALTY=1.0}"
|
||
: "${NGL:=99}"
|
||
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||
# the VRAM-fit forensics need — the exact-KV size and the draft-KV bill are
|
||
# only provable from this log. Levels above 4 flood the journal.
|
||
: "${VERBOSITY:=4}"
|
||
# /metrics needs the flag: llama.cpp returns 501 unless the server starts with
|
||
# --metrics, and Dankclaw's Prometheus scrapes every fleet endpoint's /metrics —
|
||
# a 501 reads as DOWN, which is how the critical LlmEndpointDown alert ended up
|
||
# firing for this service ever since the metric has history at all — the oldest
|
||
# retained sample of up{job="frank-zealot"} (2026-10-11) is already 0, so this
|
||
# target has never been scraped successfully, across every payload era. The
|
||
# counters cost nothing on the request path;
|
||
# METRICS=0 is the documented opt-out.
|
||
: "${METRICS:=1}"
|
||
# Vision projector. No colon on purpose: an explicitly empty value means
|
||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
||
|
||
# Fail loudly rather than half-loading: a missing file is a download that
|
||
# has not finished, and Restart=always would turn it into a crash loop that
|
||
# reads like a runtime fault.
|
||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||
done
|
||
EXTRA_ARGS=()
|
||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||
fi
|
||
|
||
# Prometheus scraping is opt-IN in llama.cpp: without --metrics the server
|
||
# answers 501 on /metrics, and the fleet scrape then reports the endpoint DOWN.
|
||
if [[ "$METRICS" = "1" ]]; then
|
||
EXTRA_ARGS+=(--metrics)
|
||
fi
|
||
|
||
# Speculative flags move as a SET: they reach the argv ONLY when a drafter is
|
||
# named, and all of them together — a no-spec baseline must not carry half a
|
||
# spec pair. n-max 2 / n-min 0 / p-min 0 is the publisher's recipe ("Draft depth
|
||
# 2 works best"); the MTP head is IN the checkpoint (blk.32.nextn.*,
|
||
# qwen35.nextn_predict_layers = 1), so there is no --spec-draft-model here and
|
||
# the draft context rides the MAIN model's weights.
|
||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE"
|
||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
|
||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN"
|
||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN")
|
||
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||
fi
|
||
fi
|
||
|
||
# The card's sampler recipe lives partly in the checkpoint: general.sampling.*
|
||
# (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0) is read by
|
||
# llama.cpp itself, and --repeat-last-n already defaults to 64. Presence penalty
|
||
# is the ONE value with no GGUF metadata key — the card says so explicitly — so
|
||
# it is the one knob that must reach the argv or the recipe is silently
|
||
# incomplete. PRESENCE_PENALTY="" is the documented opt-out.
|
||
if [[ -n "$PRESENCE_PENALTY" ]]; then
|
||
EXTRA_ARGS+=(--presence-penalty "$PRESENCE_PENALTY")
|
||
fi
|
||
|
||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||
# open after boot; make sure it exists before the server needs it.
|
||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||
|
||
# Flash attention stays pinned on. Exact f16 KV runs fine with or without it,
|
||
# but the documented fallback rung (quantized V) is ILLEGAL without it, and the
|
||
# build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17) so that rung is a
|
||
# real path — dropping the flag here would leave the retreat route silently
|
||
# falling back to something the env file does not describe.
|
||
ARGS=(
|
||
--model "$MODEL_FILE"
|
||
--alias "$ALIAS"
|
||
--host "$BIND_ADDR"
|
||
--port "$LISTEN_PORT"
|
||
--ctx-size "$CTX_SIZE"
|
||
--parallel "$PARALLEL"
|
||
--batch-size "$BATCH_SIZE"
|
||
--ubatch-size "$UBATCH_SIZE"
|
||
--n-gpu-layers "$NGL"
|
||
--flash-attn on
|
||
--cache-type-k "$CACHE_TYPE_K"
|
||
--cache-type-v "$CACHE_TYPE_V"
|
||
--log-verbosity "$VERBOSITY"
|
||
--jinja
|
||
--reasoning auto
|
||
--reasoning-format deepseek
|
||
)
|
||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|