dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144

Browse Source 
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.

MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.

Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.

contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank committed 2026-10-18 13:15:18 -07:00
1 parent 2b127163c8
commit 466fd477d8
14 files changed
+1001 -314
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/serve-frank-zealot.sh
+98 -53
View File
@@ -1,30 +1,42 @@
#!/usr/bin/env bash
# Serve prism-ml/Ternary-Bonsai-2-27B (PQ2_0 ternary quant) as frank-zealot on
# Dankclaw's RTX 3090 via llama.cpp (PrismML vendor build 10754, CUDA sm_86).
# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
# vendor build 10777, CUDA sm_86).
#
# 2026-10-17 (Frank): Bonsai-2 PQ2_0 swap at TWO 200,704-token conversations
# (--ctx-size 401408 --parallel 2) with q8_0 K + q8_0 V — f16 KV does not fit
# this geometry: the engine's own fit projected 33,198 MiB against 23,558 free
# and cudaMalloc failed on the 25,088 MiB pool. MTP stays off: this GGUF
# carries no MTP tensors (verified from its tensor-name list). Verbosity 4
# ships so the VRAM-fit forensics (KV layout, buffer sizes, graph capture)
# land in the journal.
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export at THREE
# 262,144-token conversations (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V.
# The 9B bills 17,408 B/token against the 27B's 34,816 (8 full-attention layers
# instead of 16), and that halved rate is what funds three NATIVE windows where
# the 27B could only afford two cut to 200,704. Verbosity 4 ships so the
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the
# journal.
#
# Architecture: qwen35 — 27B, 64 blocks with full_attention_interval 4 (16
# full-attention layers + 48 Gated DeltaNet linear layers), 24 heads / 4 KV
# heads × 256 dim, native 262,144, vocab 248,320. The KV pool scales with the
# 16 full layers: 34 KiB/token at q8_0 K + q8_0 V (13,328 MiB at the 401,408
# pool); the DeltaNet layers cost a fixed recurrent state per SLOT (~150 MiB —
# boot-ledger context minus KV). Boot ledger: 22,664 used / 1,461 free, with
# the Q8_0 projector's 873 MiB landing AFTER the engine's fit projection.
# The runtime is a hard lock, not a preference: the file carries 402 tensors
# of custom GGML type 142 plus prism.hadamard.* metadata, which only the
# vendor build registers — stock llama.cpp refuses it at load.
# Vision comes from the publisher's Q8_0 projector, so --mmproj is what gives
# this endpoint eyes; MMPROJ_FILE="" is the documented text-only opt-out.
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
# full layers (13,056.00 MiB at the 786,432 pool, ledger-exact); the DeltaNet
# layers cost a fixed recurrent state per SLOT — 452.25 MiB for 3 cells, ~151
# MiB each. The runtime is a hard lock, not a preference: custom GGML type 142
# plus prism.hadamard.* metadata, which only the vendor build registers, and
# MTP beside quantized V needs build ≥ b10776 — so the prebuilt moved 10754 →
# 10777 with the payload. Vision comes from the publisher's Q8_0 projector (the
# BF16 one can crash the server on concurrent images, and this endpoint runs
# three slots), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
# the documented text-only opt-out.
#
# Sampling: llama.cpp takes it per request (the fleet harness sends the file's
# own template defaults from models.yml: thinking temp 1.0 / top_p 0.95 / top_k 20).
# Speculative decoding ships OFF, measured: the head loads and accepts well
# (0.798/0.617 per position, mean length 2.41) but the draft context bills its
# own f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
# buffers — and at ubatch 2048 that squeeze HALVES decode (167.2 → 109.2 t/s).
# frank-zealot.env carries the six-arm table and the arm that turns it back on
# (--ubatch-size 1024, decode 213.5 t/s) with its price.
#
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
# repeat_penalty 1.0 — confirmed in the engine's own /props defaults) and
# llama.cpp reads it; --presence-penalty is the one recipe value with no
# metadata key, so this script passes it. A request may still override per call
# (the fleet harness sends the same values).
#
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
set -euo pipefail
@@ -39,45 +51,62 @@ if [[ -f "$ENV_FILE" ]]; then
fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf}"
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}"
: "${ALIAS:=frank-zealot}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
# per conversation, inside the model's trained 262,144. The harness
# ctx-size is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144
# per conversation — every slot at the model's trained window. The harness
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
# with a 400, never truncated.
: "${CTX_SIZE:=401408}"
: "${PARALLEL:=2}"
: "${CTX_SIZE:=786432}"
: "${PARALLEL:=3}"
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
# throughput-neutral at a fixed ubatch.
# throughput-neutral at a fixed ubatch. It ALSO sets the compute-buffer size,
# which is the term that decides whether an MTP draft context fits beside a
# 786,432-position pool: 1024 hands back 1,306 MiB and is the arm that runs the
# drafter at three slots (frank-zealot.env carries the table).
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}"
# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 2 × 200,704 on this
# card. 34 KiB/token at this geometry (16 full-attention layers × 4 KV heads ×
# 256 dim); 13,328 MiB at the 401,408 pool. f16 measured OOM; q4_0 is the
# fallback rung (16,392 used / 7,733 free). Explicit either way so the contract
# is visible in the argv, and --flash-attn stays required for quantized V
# (no true fp8 KV dtype exists in this build).
# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 3 × 262,144 on this
# card. 17 KiB/token at this geometry (8 full-attention layers × 4 KV heads ×
# 256 dim); 13,056.00 MiB at the 786,432 pool, ledger-exact. f16 would bill
# 26,112 MiB against a 24,123 MiB card and dies at cudaMalloc; q4_0 is the
# fallback rung. Explicit either way so the contract is visible in the argv,
# and --flash-attn stays required for quantized V (no true fp8 KV dtype exists
# in this build).
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
# Speculative decoding is OFF: this PQ2_0 file carries NO nextn/mtp tensors, so
# there is no in-file head (the law: MTP is a property of the file, not of the
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
# an MTP-bearing CRACK re-export exists upstream (ManniX-ITA, +451 MB) if
# decode ever needs it — that is a new payload with its own VRAM bill, never a
# flag flip on this file.
# Speculative decoding defaults OFF, measured on this card and shape: the head
# IS in the checkpoint (qwen35.nextn_predict_layers = 1 — MTP is a property of
# the FILE, so the plain PQ2_0 sibling could not run it at any flag value), it
# loads, and it accepts well (mean accepted length 2.41) — but the draft context
# bills its own f16 KV over the WHOLE pool (4,096 B/position, 3,072 MiB here)
# and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s. There is no
# --spec-draft-cache-type knob in this build to make the draft cheap, and no
# --spec-draft-model either: the draft rides the main model's weights. Card gate:
# quantized V alongside MTP needs fork build >= prism-b10776, which is why
# LLAMA_BIN moved to the 10777 prebuilt; on an older build the pair is illegal.
# SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 is the documented MTP arm (213.5 t/s);
# SPEC_TYPE=none strips the whole set (flags move together).
: "${SPEC_TYPE:=none}"
: "${SPEC_DRAFT_N_MAX:=3}"
: "${SPEC_DRAFT_N_MAX:=2}"
: "${SPEC_DRAFT_N_MIN:=0}"
: "${SPEC_DRAFT_P_MIN:=0}"
: "${SPEC_DRAFT_FILE:=}"
# No colon on purpose, same convention as MMPROJ_FILE: an explicitly empty
# PRESENCE_PENALTY is the documented opt-out and must NOT be replaced by the
# default (a colon treats null as unset and would silently re-add the flag).
: "${PRESENCE_PENALTY=1.0}"
: "${NGL:=99}"
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
# the VRAM-fit forensics need. Levels above 4 flood the journal.
: "${VERBOSITY:=3}"
# the VRAM-fit forensics need — the q8_0 KV contract and the draft-KV bill are
# only provable from this log. Levels above 4 flood the journal.
: "${VERBOSITY:=4}"
# /metrics needs the flag: llama.cpp returns 501 unless the server starts with
# --metrics, and Dankclaw's Prometheus scrapes every fleet endpoint's /metrics —
# a 501 reads as DOWN, which is how the critical LlmEndpointDown alert ended up
@@ -89,7 +118,7 @@ fi
: "${METRICS:=1}"
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf}"
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
# Fail loudly rather than half-loading: a missing file is a download that
# has not finished, and Restart=always would turn it into a crash loop that
@@ -109,15 +138,31 @@ if [[ "$METRICS" = "1" ]]; then
EXTRA_ARGS+=(--metrics)
fi
# Speculative flags move as a set: they reach the argv ONLY when a drafter
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
# baseline must not carry half a spec pair.
# Speculative flags move as a SET: they reach the argv ONLY when a drafter is
# named, and all of them together — a no-spec baseline must not carry half a
# spec pair. n-max 2 / n-min 0 / p-min 0 is the publisher's recipe ("Draft depth
# 2 works best"); the MTP head is IN the checkpoint (blk.32.nextn.*,
# qwen35.nextn_predict_layers = 1), so there is no --spec-draft-model here and
# the draft context rides the MAIN model's weights.
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE"
--spec-draft-n-max "$SPEC_DRAFT_N_MAX"
--spec-draft-n-min "$SPEC_DRAFT_N_MIN"
--spec-draft-p-min "$SPEC_DRAFT_P_MIN")
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
fi
fi
# The card's sampler recipe lives partly in the checkpoint: general.sampling.*
# (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0) is read by
# llama.cpp itself, and --repeat-last-n already defaults to 64. Presence penalty
# is the ONE value with no GGUF metadata key — the card says so explicitly — so
# it is the one knob that must reach the argv or the recipe is silently
# incomplete. PRESENCE_PENALTY="" is the documented opt-out.
if [[ -n "$PRESENCE_PENALTY" ]]; then
EXTRA_ARGS+=(--presence-penalty "$PRESENCE_PENALTY")
fi
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA