dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144

Browse Source 
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.

MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.

Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.

contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank
2026-10-18 13:15:18 -07:00
parent 2b127163c8
commit 466fd477d8
14 changed files with 1001 additions and 314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_zealot_unit.bats
+101 -52
View File
@@ -1,14 +1,17 @@
# Seam: the frank-zealot llama.cpp root unit on dankclaw.
# frank-zealot is the RTX 3090 endpoint; since 2026-10-17 it serves
# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 — the ternary Bonsai-2-27B, a
# qwen35 hybrid with 64 blocks (16 full-attention + 48 Gated DeltaNet, no
# nextn/mtp tensors in the file) — via the PrismML fork build of llama.cpp
# (vendor build 10754), the ONLY loader that registers the payload's custom
# tensor type 142 and honours its prism.hadamard.* metadata: TWO 200,704-token
# conversations (--ctx-size 401408 --parallel 2), q8_0 K + q8_0 V under flash
# attention (f16 KV does not fit this card at this pool — the boot dies on a
# 25,088 MiB cudaMalloc), the publisher's own Q8_0 vision projector, and MTP
# off. Log verbosity 4 ships so the VRAM-fit forensics land in the journal.
# frank-zealot is the RTX 3090 endpoint; since 2026-10-18 it serves
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 — the ternary Bonsai-2-9B, a
# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the
# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp
# (vendor build 10777, the minimum that pairs MTP with a quantized V cache),
# the ONLY loader that registers the payload's custom tensor type 142 and
# honours its prism.hadamard.* metadata: THREE 262,144-token conversations
# (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V under flash attention (f16
# KV would bill 26,112 MiB of a 24,123 MiB card), the publisher's own Q8_0
# vision projector (the BF16 one can crash on concurrent images), and MTP OFF
# — measured, not because the head is missing: the draft context bills f16 KV
# over the whole pool and halves decode at this shape. Log verbosity 4 ships so
# the VRAM-fit forensics land in the journal.
# This seam runs the serve script against a stub engine and asserts on the
# argv it actually execs, so what is pinned is the command the operator gets.
@@ -97,15 +100,16 @@ run_stage() {
grep -q 'FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env' "$UNIT"
}
@test "the shipped env points at the fork runtime and the Bonsai-2 PQ2_0 pair" {
@test "the shipped env points at the fork runtime and the Bonsai-2 9B MTP pair" {
# The payload path IS the endpoint's identity, and the runtime is part of
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata
# make the PrismML build the only loader, so a retarget that moves the
# model but leaves the master build behind ships an endpoint that cannot
# load at all.
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0\.gguf$' "$ENV_FILE"
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
# make the PrismML build the only loader, and this build must be >= b10776
# because MTP beside q8_0 V is illegal below it. A retarget that moves the
# model but leaves the old build behind ships an endpoint that cannot load
# (type 142) or cannot run the documented MTP arm (the build gate).
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE"
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
}
@test "the deployer gates on exactly the payload paths the env file ships" {
@@ -122,7 +126,7 @@ run_stage() {
done
}
@test "two conversations share the pool at a 200704 window each" {
@test "three conversations share the pool at a 262144 window each" {
serve_argv
[ "$status" -eq 0 ]
[ "$(argv_value --alias)" = "frank-zealot" ]
@@ -131,27 +135,31 @@ run_stage() {
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
# llama.cpp divides --ctx-size across slots, so the observable contract
# is the per-slot window. Frank's 200k cap is a per-conversation cap, so
# the pair is 401,408 / 2 — and the harness contextWindow must quote
# 200704, not the pool and not the model's native 262,144.
[ "$(argv_value --parallel)" -eq 2 ]
[ "$(argv_value --ctx-size)" -eq 401408 ]
# is the per-slot window. The 9B swap exists precisely so every
# conversation gets the model's TRAINED 262,144 (the 27B had to be cut to
# 200,704 to fund two), and the harness contextWindow must quote 262144 —
# the pool and the slot count are neither, and a longer prompt is a 400.
# Three is also the ceiling: a fourth slot would push each window under the
# fleet's worst-case compaction input, which is this endpoint's job.
[ "$(argv_value --parallel)" -eq 3 ]
[ "$(argv_value --ctx-size)" -eq 786432 ]
local pool slots
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
[ $((pool % slots)) -eq 0 ]
[ $((pool / slots)) -eq 200704 ]
[ $((pool / slots)) -eq 262144 ]
}
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
# Asserted against the env rather than a literal so a retune moves one
# file, not two. The shipped pair is q8_0/q8_0: this payload bills
# 16 full-attention layers × 4 KV heads × 256 dim = 16,384 elements per
# token per cache, so f16/f16 would need 25,088 MiB of the 24,123 MiB card
# (measured: cudaMalloc OOM, engine fit 33,198 vs 23,558 free) while q8_0
# bills 34,816 B/token = 13,328.00 MiB at the 401,408 pool. The boot
# ledger, not this file, is the referee. flash-attn is not decoration:
# quantized V is only legal with it, and it must stay explicit so a
# future quant swap cannot silently fall back.
# 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per token
# per cache, so f16/f16 would need 26,112 MiB of the 24,123 MiB card while
# q8_0 bills 17,408 B/token = 13,056.00 MiB at the 786,432 pool — the
# ledger said exactly that, and it is the referee. The 24 DeltaNet layers
# bill a fixed recurrent state per slot (452.25 MiB for three cells), which
# is why pool arithmetic alone under-prices the shape. flash-attn is not
# decoration: quantized V is only legal with it, and it must stay explicit
# so a future quant swap cannot silently fall back.
local env_k env_v
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
@@ -166,15 +174,16 @@ run_stage() {
serve_argv
[ "$status" -eq 0 ]
# The shipped 4 (debug) is what makes VRAM-fit forensics possible from
# the journal (KV layout, buffer sizes, graph capture).
# the journal (KV layout, buffer sizes, graph capture) — the only way to
# prove the q8_0 KV contract and the draft-KV bill from a log.
[ "$(argv_value --log-verbosity)" = "4" ]
# And an env that names no verbosity still ships the engine default
# explicitly — the script's written-down 3, never a blank.
# And an env that names no verbosity still ships an explicit value: the
# serve script's written-down default is 4 too, never a blank.
sed -i '/^VERBOSITY=/d' "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
[ "$(argv_value --log-verbosity)" = "3" ]
[ "$(argv_value --log-verbosity)" = "4" ]
}
@test "the Prometheus metrics endpoint is served unless opted out" {
@@ -221,12 +230,16 @@ run_stage() {
}
@test "speculation is off: no drafter flag reaches the argv" {
# Frank, 2026-10-07: MTP is off because it CANNOT be on — the Q6_K file
# carries no nextn/mtp tensors (verified from its tensor-name list), and
# MTP is a property of the file, not of the flags. The flags move as a
# set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
# --spec-draft-model together, so a no-spec boot cannot carry half a
# spec pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
# Frank, 2026-10-18: this file DOES carry the MTP head (the boot arms prove
# it loads and drafts, acceptance 0.798/0.617 per position, mean accepted
# length 2.41) and it is still off, because the draft context bills its own
# f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
# buffers — and at --ubatch-size 2048 that squeeze halves decode: 167.2 →
# 109.2 t/s. The MTP arm is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (213.5
# t/s), documented in frank-zealot.env. The flags move as a set:
# SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
# --spec-draft-model together, so a no-spec boot cannot carry half a spec
# pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
serve_argv
[ "$status" -eq 0 ]
! grep -qxF -- '--spec-type' <<<"$output"
@@ -235,27 +248,35 @@ run_stage() {
grep -qE '^SPEC_TYPE=none$' "$ENV_FILE"
}
@test "naming a drafter ships the full MTP trio" {
# The opt-back-in path (e.g. a future Qwen3.5 -mtp build): SPEC_TYPE +
# SPEC_DRAFT_FILE must reach the engine as a complete set — a draft-mtp
# flag without its model file boots a drafter against the main weights and
# drafts nothing.
@test "naming a drafter ships the MTP set, with or without a separate draft file" {
# The documented MTP arm (frank-zealot.env): SPEC_TYPE=draft-mtp ALONE is
# legal here because the head is IN the checkpoint and the draft context
# rides the main model's weights — so --spec-draft-model must NOT appear,
# or the trunk would be loaded twice.
serve_argv
{
echo 'SPEC_TYPE=draft-mtp'
echo 'SPEC_DRAFT_N_MAX=3'
echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB"
echo 'SPEC_DRAFT_N_MAX=2'
} >> "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
[ "$(argv_value --spec-type)" = "draft-mtp" ]
[ "$(argv_value --spec-draft-n-max)" = "3" ]
[ "$(argv_value --spec-draft-n-max)" = "2" ]
! grep -qxF -- '--spec-draft-model' <<<"$output"
# And an external drafter still reaches the engine as a complete set — the
# pair the ROCmFPX-era crash was about.
echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB" >> "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
[ "$(argv_value --spec-draft-model)" = "$MODEL_DIR/$DRAFT_STUB" ]
}
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
# Qwen3.5-9B is natively multimodal, but the Q6_K file carries no vision
# tensors — the publisher's projector argument IS the vision capability:
# no flag, no eyes.
# The Bonsai-2 text file carries no vision tensors — the publisher's Q8_0
# projector argument IS the vision capability: no flag, no eyes. Q8_0 (not
# BF16) is the publisher's own requirement for concurrent image requests,
# which is what --parallel 3 means in practice.
serve_argv
[ "$status" -eq 0 ]
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
@@ -268,6 +289,34 @@ run_stage() {
! grep -qxF -- '--mmproj' <<<"$output"
}
@test "the card sampler's only non-metadata term reaches the argv" {
# general.sampling.* (temperature / top_p / top_k / min_p / repeat_penalty)
# rides in the checkpoint and the engine reads it itself; presence penalty
# is the ONE recipe value with no GGUF metadata key, so it must reach argv
# or the recipe is silently incomplete. Distinctive value first: proof the
# flag is wired to the env and not merely to the script's default.
serve_argv
{ echo 'PRESENCE_PENALTY=0.5'; } >> "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
[ "$(argv_value --presence-penalty)" = "0.5" ]
# The shipped recipe value reaches the engine too.
local pp
pp="$(sed -nE 's/^PRESENCE_PENALTY=(.+)$/\1/p' "$ENV_FILE")"
sed -i "s/^PRESENCE_PENALTY=.*/PRESENCE_PENALTY=$pp/" "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
[ "$(argv_value --presence-penalty)" = "$pp" ]
# The documented opt-out is absence of the flag, not an empty argument —
# which is why the default uses ${VAR=default}, not ${VAR:=default}.
sed -i "s/^PRESENCE_PENALTY=.*/PRESENCE_PENALTY=/" "$STAGE/test.env"
run_stage
[ "$status" -eq 0 ]
! grep -qxF -- '--presence-penalty' <<<"$output"
}
@test "the fleet reasoning contract reaches the argv" {
serve_argv
[ "$status" -eq 0 ]