Frank
2b127163c8
frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs one 401,408-position pool split into two 200,704 conversations, while dragoon ran ONE 200,704-position pool behind its two slots, so a single maximum-length request filled it and two long conversations had to stay under ~100k each. HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move: HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What this buys is concurrency, not a longer conversation, which is why the harness contextWindow (155648 = ceiling - 45056) is untouched. Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the 2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB n-gram table reads through — the warm prompt-cache ratio is the number to re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2. Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with the publisher's own throughput table in its header — the packing is smaller but the card measures it SLOWER on Ampere, so the swap is decided on our card, not on the card's A100 row.
125 lines
4.9 KiB
Bash
Executable File
125 lines
4.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Candidate payload for frank-zealot: prism-ml/Ternary-Bonsai-2-27B-gguf at
|
|
# **PTQ1_0** — the DENSE ternary packing (1.75 bits/weight, 5.95 GB) of the same
|
|
# weights the live PQ2_0 endpoint serves (2.13 bits/weight, 7.21 GB). Public
|
|
# repo: no auth header, and never a token on the command line (a Bearer token is
|
|
# world-readable in /proc/<pid>/cmdline).
|
|
#
|
|
# WHY THIS ARM EXISTS. The whole argument is footprint: 5,946,648,928 B against
|
|
# the live 7,206,168,928 B = **1,201 MiB freed on the card**. On this endpoint
|
|
# KV bills 34,816 B/token (16 full-attention layers x 4 KV heads x 256 dim at
|
|
# q8_0 K + q8_0 V), so that one gap is ~37,000 tokens of pool per 1,201 MiB —
|
|
# context is exactly what the freed weight bytes buy.
|
|
#
|
|
# WHY IT MIGHT LOSE. The publisher's own throughput table (README, llama-bench,
|
|
# batch 1, depth 0): PTQ1_0 wins decode only on Ada/L4 (RTX 4090 91.1 vs 81.2,
|
|
# RTX 6000 Ada 90.4 vs 82.8, L4 32.1 vs 29.8) and LOSES on H100 (86.9 vs 113.9)
|
|
# and A100 (54.7 vs 73.9); prompt processing "favors PQ2_0 everywhere" (A100
|
|
# 706 vs 1328). The card text says explicitly that dense-trit unpacking costs
|
|
# arithmetic and the pack is "slower on Ampere, Hopper, and Blackwell". There is
|
|
# NO GA102/RTX 3090 row: the Ampere statement generalizes from an A100, which is
|
|
# instruction-bound at batch 1 with 2 TB/s of HBM, while the 3090 is the
|
|
# bandwidth-starved member of the family (936 GB/s) — the one regime where
|
|
# moving 17 % less weight per step can pay. That is an argument, not a
|
|
# measurement: this script exists so the decision is made on OUR card, against
|
|
# the PQ2_0 record (76.6 t/s single-stream decode, 1,258 t/s prefill at 35k).
|
|
#
|
|
# THE RUNTIME LOCK CARRIES OVER. Both packings are custom GGML tensor types and
|
|
# stock llama.cpp rejects them ("It rejects PQ2_0 and PTQ1_0 as unknown types"
|
|
# — the card), so the PrismML fork build stays mandatory. Confirm the loadability
|
|
# against the installed binary BEFORE pointing any unit at this file.
|
|
#
|
|
# The vision projector is the SAME file the PQ2_0 endpoint already uses (Hub LFS
|
|
# oid 6807ede61d57… matches byte-for-byte), so it is HARDLINKED from the existing
|
|
# payload dir instead of re-downloaded — a hardlink is a regular file, so the
|
|
# serve gates' no-symlink rule stays satisfied.
|
|
#
|
|
# Usage:
|
|
# scripts/download-frank-zealot-bonsai2-ptq1.sh [DEST_DIR] [--verify-only]
|
|
set -euo pipefail
|
|
|
|
DEST=""
|
|
VERIFY_ONLY=0
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--verify-only) VERIFY_ONLY=1 ;;
|
|
*) DEST="$arg" ;;
|
|
esac
|
|
done
|
|
DEST="${DEST:-/home/dank/models/Bonsai2-PTQ1_0}"
|
|
LINK_SRC="${LINK_SRC:-/home/dank/models/Bonsai2-PQ2_0}"
|
|
|
|
REPO="prism-ml/Ternary-Bonsai-2-27B-gguf"
|
|
MMPROJ="Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf"
|
|
MMPROJ_SIZE=629246976
|
|
MMPROJ_SHA="6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903"
|
|
|
|
# hub-name | dest-name | bytes | sha256 (Hub LFS oid)
|
|
FILES=(
|
|
"Ternary-Bonsai-2-27B-PTQ1_0.gguf|Ternary-Bonsai-2-27B-PTQ1_0.gguf|5946648928|53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3"
|
|
)
|
|
|
|
mkdir -p "$DEST"
|
|
status=0
|
|
|
|
check() { # file size sha -> 0 when byte- and hash-exact
|
|
[[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
|
|
&& sha256sum -c --quiet <(printf '%s %s\n' "$3" "$1")
|
|
}
|
|
|
|
for entry in "${FILES[@]}"; do
|
|
IFS='|' read -r hubname destname size sha <<<"$entry"
|
|
f="$DEST/$destname"
|
|
if check "$f" "$size" "$sha"; then
|
|
echo "OK $destname"
|
|
continue
|
|
fi
|
|
if (( VERIFY_ONLY )); then
|
|
echo "MISSING $destname"
|
|
status=1
|
|
continue
|
|
fi
|
|
echo "FETCHING $destname (from $REPO)"
|
|
tmp="$f.part"
|
|
# No resume: `curl -C -` once produced an oversized file on a 31 GiB payload.
|
|
# -f/--fail-with-body so an HTTP 401/403 body is never written as success.
|
|
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
|
-o "$tmp" "https://huggingface.co/${REPO}/resolve/main/${hubname}"
|
|
mv "$tmp" "$f"
|
|
if ! check "$f" "$size" "$sha"; then
|
|
echo "FAIL $destname (size $(stat -c%s "$f" 2>/dev/null || echo none) != $size, or sha mismatch)"
|
|
status=1
|
|
continue
|
|
fi
|
|
echo "VERIFIED $destname"
|
|
done
|
|
|
|
# Vision projector: hardlink the identical file when a verified copy exists.
|
|
mm="$DEST/$MMPROJ"
|
|
if check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
|
echo "OK $MMPROJ"
|
|
elif [[ -f "$LINK_SRC/$MMPROJ" ]] && check "$LINK_SRC/$MMPROJ" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
|
ln "$LINK_SRC/$MMPROJ" "$mm" && echo "HARDLINKED $MMPROJ from $LINK_SRC"
|
|
elif (( VERIFY_ONLY )); then
|
|
echo "MISSING $MMPROJ"
|
|
status=1
|
|
else
|
|
echo "FETCHING $MMPROJ (no verified source copy to link)"
|
|
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
|
-o "$mm.part" "https://huggingface.co/${REPO}/resolve/main/${MMPROJ}"
|
|
mv "$mm.part" "$mm"
|
|
if ! check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
|
echo "FAIL $MMPROJ"
|
|
status=1
|
|
else
|
|
echo "VERIFIED $MMPROJ"
|
|
fi
|
|
fi
|
|
|
|
if (( status == 0 )); then
|
|
echo "DOWNLOAD_DONE $DEST"
|
|
else
|
|
echo "DOWNLOAD_FAILED $DEST" >&2
|
|
fi
|
|
exit "$status"
|