Frank
466fd477d8
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
88 lines
3.1 KiB
Bash
Executable File
88 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Download the frank-zealot payload for llama.cpp on the RTX 3090:
|
|
# prism-ml/Ternary-Bonsai-2-9B-gguf — the **PQ2_0 packing with the MTP draft
|
|
# head built in** (the card: "load it instead of the plain PQ2_0, not alongside
|
|
# it") plus the PUBLISHER's OWN **Q8_0** vision tower. Public repo — no auth
|
|
# header, and never a token on the command line (world-readable in
|
|
# /proc/<pid>/cmdline).
|
|
#
|
|
# THE Q8_0 mmproj IS MANDATORY, NOT A PREFERENCE. The card: with the current
|
|
# fork build "the BF16 mmproj can crash llama-server when several image
|
|
# requests run at once; Q8_0 is not affected and quality is nearly the same."
|
|
# frank-zealot runs --parallel > 1, i.e. exactly that concurrent-image shape.
|
|
#
|
|
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file
|
|
# on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a success,
|
|
# then byte count and sha256 against the Hub's **LFS oid** (this repo IS LFS,
|
|
# unlike the HauhauCS Xet repo the 2026-10-07 era used, so the sha values below
|
|
# are the Hub's own — verified at fetch, not computed locally).
|
|
# Idempotent: complete files are skipped.
|
|
#
|
|
# Rollback payload stays on disk: /home/dank/models/Bonsai2-PQ2_0/ (the
|
|
# Ternary-Bonsai-2-27B PQ2_0 file + its mmproj, cut over 2026-10-17), with
|
|
# Bonsai2-CRACK/ (the uncensored 27B re-export) beside it.
|
|
#
|
|
# Usage:
|
|
# scripts/download-frank-zealot-bonsai9.sh [DEST_DIR] [--verify-only]
|
|
set -euo pipefail
|
|
|
|
DEST=""
|
|
VERIFY_ONLY=0
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--verify-only) VERIFY_ONLY=1 ;;
|
|
*) DEST="$arg" ;;
|
|
esac
|
|
done
|
|
DEST="${DEST:-/home/dank/models/Bonsai2-9B-MTP}"
|
|
|
|
# repo | hub-name | dest-name | bytes | sha256 (Hub LFS oid)
|
|
FILES=(
|
|
"prism-ml/Ternary-Bonsai-2-9B-gguf|Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf|Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf|2675410720|c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b"
|
|
"prism-ml/Ternary-Bonsai-2-9B-gguf|Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf|Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf|624229664|e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db"
|
|
)
|
|
|
|
mkdir -p "$DEST"
|
|
status=0
|
|
for entry in "${FILES[@]}"; do
|
|
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
|
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
|
|
f="$DEST/$destname"
|
|
have=0
|
|
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
|
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
|
have=1
|
|
fi
|
|
fi
|
|
if (( have )); then
|
|
echo "OK $destname"
|
|
continue
|
|
fi
|
|
if (( VERIFY_ONLY )); then
|
|
echo "MISSING $destname"
|
|
status=1
|
|
continue
|
|
fi
|
|
echo "FETCHING $destname (from $repo)"
|
|
tmp="$f.part"
|
|
curl -fsSL --fail-with-body -o "$tmp" "$url"
|
|
mv "$tmp" "$f"
|
|
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
|
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
|
|
status=1
|
|
continue
|
|
fi
|
|
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
|
echo "SHA FAIL $destname"
|
|
status=1
|
|
continue
|
|
fi
|
|
echo "VERIFIED $destname"
|
|
done
|
|
if (( status == 0 )); then
|
|
echo "DOWNLOAD_DONE $DEST"
|
|
else
|
|
echo "DOWNLOAD_FAILED $DEST" >&2
|
|
fi
|
|
exit "$status"
|