frank-large: correct the halogen payload contract to the Hub's truth

The first constants were wrong in three ways, found while resuming the
checkpoint (the file was COMPLETE; the manifest was not):
- checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not
  35,880,329,102 / 40874e7a...), 1352 tensors per the card;
- the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole
  at /tokenizer (model-card docker-run parity; the engine's
  HALOGEN_TOKENIZER default is the directory) — there is no
  frank-large.tokenizer.gguf;
- plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the
  fetch script size-checks them and sha-verifies only the two LFS
  objects (checkpoint, tokenizer.json).

Gates flipped together (serve + deployer twins): tokenizer dir must be a
real directory and every named file a regular file, no symlinks (HF
snapshot links dangle in the mount). Fetch made idempotent: a file at
the published size skips the transfer (resuming it is a CDN 416 under
set -e). Seam rewritten to the same contract, 17/17.
This commit is contained in:
Frank committed 2026-10-05 16:45:12 -07:00
1 parent 87695c5126
commit bdbe814fac
7 files changed
+126 -66

No files matched your search

+37 -16
View File
@@ -13,12 +13,19 @@ set -euo pipefail
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
FILES=(qwen3.8-27b-p1w4d-d2.hgn frank-large.tokenizer.gguf)
# Byte sizes and sha256 (LFS oid) as published by the Hub on 2026-10-05 —
# re-check with: curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
SIZES=(35880329102 1293343)
SHAS=(40874e7a5c27efcf1c83c44374db85726fa4086355b3c8aae1b363644d7e6234
4c1ab3a19932f7d16eefc2c7fa60715c91133fe99c6dce683a3a7f11d463e7e9)
FILES=(qwen3.8-27b-p1w4d-d2.hgn
tokenizer/tokenizer.json
tokenizer/vocab.json
tokenizer/merges.txt
tokenizer/chat_template.jinja
tokenizer/tokenizer_config.json)
# Byte sizes as published by the Hub on 2026-10-05, plus sha256 (the LFS
# oid) where the Hub stores an LFS object — plain git blobs carry no oid
# and are size-checked only. Re-check with:
# curl -s "https://huggingface.co/api/models/peonist-ai/halogen-qwen3.8-27b/tree/main?recursive=true"
SIZES=(35865565184 12809320 6722759 3353259 8952 17928)
SHAS=(274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3 - - - -)
VERIFY_ONLY=0
[ "${1:-}" = "--verify-only" ] && VERIFY_ONLY=1
@@ -28,16 +35,26 @@ mkdir -p "$DEST"
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; want="${SIZES[$i]}"; sha="${SHAS[$i]}"
path="$DEST/$f"
mkdir -p "$(dirname "$path")"
if [ "$VERIFY_ONLY" -eq 0 ]; then
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS the
# Range header across redirects — a naive -C - then "resumes" by
# pulling the ~1 KB redirect body (measured 2026-10-05: file frozen
# 14 MiB short, curl exit 0). Resolve the effective URL first, then
# resume against the CDN, which honors Range.
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
|| { echo "cannot resolve $BASE/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
# A file already at the published size skips the fetch entirely:
# resuming it would ask the CDN for a range past EOF (416 -> curl
# error under set -e), and re-pulling 33.4 GiB to prove a point is
# not what --verify-only is for. sha256 below still decides.
if [ -f "$path" ] && [ "$(stat -c %s "$path")" = "$want" ]; then
echo "HAVE $f ($want bytes) — skipping fetch"
else
echo "FETCH $f -> $path"
# HF answers /resolve/ with a 302 to the Xet CDN and curl DROPS
# the Range header across redirects — a naive -C - then
# "resumes" by pulling the ~1 KB redirect body (measured
# 2026-10-05: file frozen 14 MiB short, curl exit 0). Resolve
# the effective URL first, then resume against the CDN, which
# honors Range.
url=$(curl -sfIL -o /dev/null -w '%{url_effective}' "$BASE/$f") \
|| { echo "cannot resolve $BASE/$f" >&2; exit 1; }
curl -fL --retry 5 -C - -o "$path" "$url"
fi
fi
[ -f "$path" ] || { echo "MISSING $path (run without --verify-only)" >&2; exit 1; }
got="$(stat -c %s "$path")"
@@ -45,9 +62,13 @@ for i in "${!FILES[@]}"; do
echo "OK $f ($got bytes)"
done
echo "byte counts match; verifying sha256 (one pass over 33.4 GiB)..."
echo "byte counts match; verifying sha256 where the Hub publishes an LFS oid..."
for i in "${!FILES[@]}"; do
f="${FILES[$i]}"; sha="${SHAS[$i]}"
if [ "$sha" = "-" ]; then
echo "OK $f (plain git blob, size-checked)"
continue
fi
got="$(sha256sum "$DEST/$f" | cut -d' ' -f1)"
[ "$got" = "$sha" ] || { echo "SHA256 $f: $got != $sha" >&2; exit 1; }
echo "OK $f sha256"