Frank
06a9cb60eb
Frank's 02:36 instruction (archive 01a14c93) ordered the 27B PTQ1_0 arm at 2 x 200,704 with the Q8 mmproj and batch/ubatch 3072/1536; it served production 03:14:39-03:38:57 (pid 191358, journal type ptq1_0: 402 tensors). Recovered its numbers from that process's own print_timing lines: ~48 t/s single-stream decode, 25-27 t/s per stream at two streams, prefill 1,295 t/s at 30,637 -> 943 at 97,856 -> 801 at 141,365. It bought the promised 1,201 MiB and lost 37% of decode against the PQ2_0 record, so the publisher's Ampere warning generalizes to GA102; Frank scratched the 27B plan at 03:33 for the 9B. Every 'unfinished experiment' claim is re-keyed, and the AGENTS.md law gains the durable lesson: a bench verdict that never reaches the repo is indistinguishable from a rogue arm.
134 lines
5.6 KiB
Bash
Executable File
134 lines
5.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Candidate payload for frank-zealot: prism-ml/Ternary-Bonsai-2-27B-gguf at
|
||
# **PTQ1_0** — the DENSE ternary packing (1.75 bits/weight, 5.95 GB) of the same
|
||
# weights the live PQ2_0 endpoint serves (2.13 bits/weight, 7.21 GB). Public
|
||
# repo: no auth header, and never a token on the command line (a Bearer token is
|
||
# world-readable in /proc/<pid>/cmdline).
|
||
#
|
||
# WHY THIS ARM EXISTS. The whole argument is footprint: 5,946,648,928 B against
|
||
# the live 7,206,168,928 B = **1,201 MiB freed on the card**. On this endpoint
|
||
# KV bills 34,816 B/token (16 full-attention layers x 4 KV heads x 256 dim at
|
||
# q8_0 K + q8_0 V), so that one gap is ~37,000 tokens of pool per 1,201 MiB —
|
||
# context is exactly what the freed weight bytes buy.
|
||
#
|
||
# WHY IT MIGHT LOSE. The publisher's own throughput table (README, llama-bench,
|
||
# batch 1, depth 0): PTQ1_0 wins decode only on Ada/L4 (RTX 4090 91.1 vs 81.2,
|
||
# RTX 6000 Ada 90.4 vs 82.8, L4 32.1 vs 29.8) and LOSES on H100 (86.9 vs 113.9)
|
||
# and A100 (54.7 vs 73.9); prompt processing "favors PQ2_0 everywhere" (A100
|
||
# 706 vs 1328). The card text says explicitly that dense-trit unpacking costs
|
||
# arithmetic and the pack is "slower on Ampere, Hopper, and Blackwell". There is
|
||
# NO GA102/RTX 3090 row: the Ampere statement generalizes from an A100, which is
|
||
# instruction-bound at batch 1 with 2 TB/s of HBM, while the 3090 is the
|
||
# bandwidth-starved member of the family (936 GB/s) — the one regime where
|
||
# moving 17 % less weight per step can pay. That is an argument, not a
|
||
# measurement: this script exists so the decision is made on OUR card, against
|
||
# the PQ2_0 record (76.6 t/s single-stream decode, 1,258 t/s prefill at 35k).
|
||
# MEASURED ON OUR CARD (2026-10-18). Frank ordered this shape at 02:36 — Q8
|
||
# mmproj, batch 3072/ubatch 1536, two contexts — and the unit ran it as
|
||
# production 03:14:39→03:38:57 (pid 191358, journal `type ptq1_0: 402 tensors`).
|
||
# Its own print_timing lines: decode **48.0 / 47.6 t/s at 512 tokens** (47.9 at
|
||
# 229, 49.0 at 451), 25–27 t/s per stream with two streams live, prefill 1,295
|
||
# t/s at 30,637 → 943 at 97,856 → 801 at 141,365. The 1,201 MiB arrived and the
|
||
# decode tax came with it: −37 % against the PQ2_0 record, so the Ampere warning
|
||
# DOES generalize to GA102. Frank scratched the 27B plan at 03:33 for the 9B;
|
||
# this payload is the 27B rollback, not a candidate.
|
||
#
|
||
# THE RUNTIME LOCK CARRIES OVER. Both packings are custom GGML tensor types and
|
||
# stock llama.cpp rejects them ("It rejects PQ2_0 and PTQ1_0 as unknown types"
|
||
# — the card), so the PrismML fork build stays mandatory. Confirm the loadability
|
||
# against the installed binary BEFORE pointing any unit at this file.
|
||
#
|
||
# The vision projector is the SAME file the PQ2_0 endpoint already uses (Hub LFS
|
||
# oid 6807ede61d57… matches byte-for-byte), so it is HARDLINKED from the existing
|
||
# payload dir instead of re-downloaded — a hardlink is a regular file, so the
|
||
# serve gates' no-symlink rule stays satisfied.
|
||
#
|
||
# Usage:
|
||
# scripts/download-frank-zealot-bonsai2-ptq1.sh [DEST_DIR] [--verify-only]
|
||
set -euo pipefail
|
||
|
||
DEST=""
|
||
VERIFY_ONLY=0
|
||
for arg in "$@"; do
|
||
case "$arg" in
|
||
--verify-only) VERIFY_ONLY=1 ;;
|
||
*) DEST="$arg" ;;
|
||
esac
|
||
done
|
||
DEST="${DEST:-/home/dank/models/Bonsai2-PTQ1_0}"
|
||
LINK_SRC="${LINK_SRC:-/home/dank/models/Bonsai2-PQ2_0}"
|
||
|
||
REPO="prism-ml/Ternary-Bonsai-2-27B-gguf"
|
||
MMPROJ="Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf"
|
||
MMPROJ_SIZE=629246976
|
||
MMPROJ_SHA="6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903"
|
||
|
||
# hub-name | dest-name | bytes | sha256 (Hub LFS oid)
|
||
FILES=(
|
||
"Ternary-Bonsai-2-27B-PTQ1_0.gguf|Ternary-Bonsai-2-27B-PTQ1_0.gguf|5946648928|53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3"
|
||
)
|
||
|
||
mkdir -p "$DEST"
|
||
status=0
|
||
|
||
check() { # file size sha -> 0 when byte- and hash-exact
|
||
[[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
|
||
&& sha256sum -c --quiet <(printf '%s %s\n' "$3" "$1")
|
||
}
|
||
|
||
for entry in "${FILES[@]}"; do
|
||
IFS='|' read -r hubname destname size sha <<<"$entry"
|
||
f="$DEST/$destname"
|
||
if check "$f" "$size" "$sha"; then
|
||
echo "OK $destname"
|
||
continue
|
||
fi
|
||
if (( VERIFY_ONLY )); then
|
||
echo "MISSING $destname"
|
||
status=1
|
||
continue
|
||
fi
|
||
echo "FETCHING $destname (from $REPO)"
|
||
tmp="$f.part"
|
||
# No resume: `curl -C -` once produced an oversized file on a 31 GiB payload.
|
||
# -f/--fail-with-body so an HTTP 401/403 body is never written as success.
|
||
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
||
-o "$tmp" "https://huggingface.co/${REPO}/resolve/main/${hubname}"
|
||
mv "$tmp" "$f"
|
||
if ! check "$f" "$size" "$sha"; then
|
||
echo "FAIL $destname (size $(stat -c%s "$f" 2>/dev/null || echo none) != $size, or sha mismatch)"
|
||
status=1
|
||
continue
|
||
fi
|
||
echo "VERIFIED $destname"
|
||
done
|
||
|
||
# Vision projector: hardlink the identical file when a verified copy exists.
|
||
mm="$DEST/$MMPROJ"
|
||
if check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||
echo "OK $MMPROJ"
|
||
elif [[ -f "$LINK_SRC/$MMPROJ" ]] && check "$LINK_SRC/$MMPROJ" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||
ln "$LINK_SRC/$MMPROJ" "$mm" && echo "HARDLINKED $MMPROJ from $LINK_SRC"
|
||
elif (( VERIFY_ONLY )); then
|
||
echo "MISSING $MMPROJ"
|
||
status=1
|
||
else
|
||
echo "FETCHING $MMPROJ (no verified source copy to link)"
|
||
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
||
-o "$mm.part" "https://huggingface.co/${REPO}/resolve/main/${MMPROJ}"
|
||
mv "$mm.part" "$mm"
|
||
if ! check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||
echo "FAIL $MMPROJ"
|
||
status=1
|
||
else
|
||
echo "VERIFIED $MMPROJ"
|
||
fi
|
||
fi
|
||
|
||
if (( status == 0 )); then
|
||
echo "DOWNLOAD_DONE $DEST"
|
||
else
|
||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||
fi
|
||
exit "$status"
|