frank-primary cutover: tcclaviger RDNA4 stack, 600k ctx / 3 sessions / q8-q8 KV
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
This commit is contained in:
Frank committed
1 parent
9cf4b1de82
commit
3bb0e82ccf
6 files changed
+409
-389
No files matched your search
@@ -1,92 +1,88 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the frank-primary payload: the Qwen3.8-Flash-Next Q8_0 target (six
|
||||
# split parts), the shared Q8_0 MTP drafter head and the BF16 vision
|
||||
# projector.
|
||||
# Payload inventory/verification for frank-primary:
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
|
||||
# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
|
||||
# became frank-primary on 2026-09-29.
|
||||
#
|
||||
# bash scripts/download-frank-primary.sh
|
||||
# bash scripts/download-frank-primary.sh --verify-only
|
||||
# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode)
|
||||
# bash scripts/download-frank-primary.sh --fetch
|
||||
#
|
||||
# The repo (unsloth/Qwen3.8-Flash-Next-GGUF) is PUBLIC and anonymous: no HF
|
||||
# token is needed, and none is accepted on the command line (a -H auth
|
||||
# header is world-readable in /proc/<pid>/cmdline -- the 2026-09-16 leak).
|
||||
# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
|
||||
# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
|
||||
# manifest cannot be built without the authed endpoint. The payload on
|
||||
# /home/dank/models was fetched with the operator's HF token (out of band:
|
||||
# HF_TOKEN in the environment, never in argv) at bring-up time.
|
||||
#
|
||||
# WHY THESE FILES:
|
||||
# * Target: Q8_0/ -- unsloth's Q8_0 of Qwen3.8-Flash-Next (125B total /
|
||||
# 6B active MoE + the 51B n-gram embedding table). Q_8 is the
|
||||
# operator's requested quant (2026-09-26 swap off the lucebox DS4.1
|
||||
# engine). The six parts ARE the payload: part 1's header advertises
|
||||
# file_count=6, so they are never concatenated (concatenation trips
|
||||
# "invalid split file name").
|
||||
# * Drafter: MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -- the shared
|
||||
# MTP head (1 layer). "shared" means it borrows the target's embedding
|
||||
# tensors instead of carrying its own copy: 2.6 GiB instead of 3.9,
|
||||
# and unsloth's own MTP guide names it as the recommended head. It is
|
||||
# loaded via -md (this build never autoloads drafters or projectors).
|
||||
# * Vision: mmproj-BF16.gguf -- the BF16 projector, RDNA4-native per the
|
||||
# frank-small precedent. Loaded via --mmproj explicitly.
|
||||
#
|
||||
# The script is resumable (curl -C -) and checks every artifact's exact byte
|
||||
# count AND sha256 against the Hub's declared values, so a truncated or
|
||||
# mis-fetched file fails here instead of load-crashing the server. Sizes and
|
||||
# hashes were captured from the resolve endpoint's x-linked-etag/x-linked-size
|
||||
# on 2026-09-26.
|
||||
#
|
||||
# Total: 191,919,144,448 bytes (178.74 GiB), of which 175.30 GiB is the
|
||||
# target itself. Destination keeps the Hub layout so the serve script can
|
||||
# name paths exactly as they appear upstream.
|
||||
# Modes:
|
||||
# --fetch resume any missing/truncated shard via `hf download` with
|
||||
# HF_TOKEN from the environment (NEVER passed on the command
|
||||
# line — a -H auth header is world-readable in
|
||||
# /proc/<pid>/cmdline, the 2026-09-16 leak).
|
||||
# --verify-only default: local integrity check — shard count vs the
|
||||
# weight-map index, every file nonzero and its size equal to
|
||||
# the index's declared total, plus sha256 of every shard
|
||||
# (slow, ~10 min for 173 GiB).
|
||||
set -euo pipefail
|
||||
|
||||
REPO="unsloth/Qwen3.8-Flash-Next-GGUF"
|
||||
DEST="/home/dank/models/Qwen3.8-Flash-Next-GGUF"
|
||||
REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
EXPECTED_SHARDS=131
|
||||
|
||||
# path|bytes|sha256
|
||||
MANIFEST=(
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf|10946624|2dabcbb53ca537a7947bc7d20414fd464eeaf4d66d43021b5b2556cc87544ad2"
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf|682434912|494ca4ed3dbf97bc28da88af3890b8877b9032f909812d00c0526a9ca5e91d2e"
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf|54400261312|34efd79a80a1ce540a517a5d56171924b66ce1c38b04c904f17ad6d8ef17cf20"
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf|49446841216|bfa634025fabbd2658bf7694bc80b90e571699c768723f844c934c7ef06c691a"
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf|49668930400|232a8f14cc0fa4262e7efe8593774b136fe40909e39c7a020342ddaa27259a97"
|
||||
"Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf|34015618784|538a93bca918064983409a41187ad4c68640f9aced6f29564da8f551bf86d7a5"
|
||||
"MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf|2786568256|5ff54097406a905cf3a724c709124ceb0e3e10235ee862298969e91c96fa96e6"
|
||||
"mmproj-BF16.gguf|907542944|2e788f8c511d8093c7b43cb87b2fd7e14228340318057f8fb20c86df2efe2355"
|
||||
)
|
||||
mode="verify"
|
||||
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||
|
||||
mode="download"
|
||||
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
||||
|
||||
if [[ "$mode" == "download" ]]; then
|
||||
for entry in "${MANIFEST[@]}"; do
|
||||
path="${entry%%|*}"
|
||||
mkdir -p "$DEST/$(dirname "$path")"
|
||||
echo "fetching $path"
|
||||
# -f: a 401/403 must FAIL the fetch, not write the error body as
|
||||
# the artifact (the 2026-09-16 gated-repo incident). -C -: resume
|
||||
# partial files.
|
||||
curl -fL -C - -o "$DEST/$path" --retry 10 --retry-delay 5 \
|
||||
--retry-max-time 21600 \
|
||||
"https://huggingface.co/$REPO/resolve/main/$path"
|
||||
done
|
||||
if [[ "$mode" == "fetch" ]]; then
|
||||
: "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
|
||||
export HF_TOKEN
|
||||
mkdir -p "$DEST"
|
||||
hf download "$REPO" --local-dir "$DEST"
|
||||
fi
|
||||
|
||||
echo "verifying sizes and sha256"
|
||||
ok=1
|
||||
for entry in "${MANIFEST[@]}"; do
|
||||
path="${entry%%|*}"; rest="${entry#*|}"
|
||||
size="${rest%%|*}"; sha256="${rest#*|}"
|
||||
file="$DEST/$path"
|
||||
act="$(stat -c%s "$file" 2>/dev/null || echo 0)"
|
||||
if [[ "$act" -ne "$size" ]]; then
|
||||
echo " BAD-SIZE $file ($act bytes, want $size)" >&2
|
||||
ok=0
|
||||
continue
|
||||
fi
|
||||
act_sha="$(sha256sum "$file" | cut -d' ' -f1)"
|
||||
if [[ "$act_sha" != "$sha256" ]]; then
|
||||
echo " BAD-SHA $file ($act_sha)" >&2
|
||||
ok=0
|
||||
continue
|
||||
fi
|
||||
echo " OK $path ($act bytes)"
|
||||
done
|
||||
[[ "$ok" -eq 1 ]] || exit 1
|
||||
echo "all artifacts present, sized and hashed correctly"
|
||||
echo "verifying local payload: $DEST"
|
||||
fail=0
|
||||
|
||||
# The weight-map index is the source of truth for the shard set.
|
||||
INDEX="$DEST/model.safetensors.index.json"
|
||||
if [[ ! -f "$INDEX" ]]; then
|
||||
echo " BAD weight-map index missing: $INDEX" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Every index entry must exist, nonzero, and sized equal to the map's total.
|
||||
python3 - "$INDEX" "$DEST" <<'PY' || fail=1
|
||||
import json, os, sys
|
||||
index, dest = sys.argv[1], sys.argv[2]
|
||||
wm = json.load(open(index))["weight_map"]
|
||||
files = sorted(set(wm.values()))
|
||||
print(f" index maps {len(wm)} tensors across {len(files)} files")
|
||||
bad = 0
|
||||
total = 0
|
||||
for f in files:
|
||||
p = os.path.join(dest, f)
|
||||
if not os.path.isfile(p) or os.path.getsize(p) == 0:
|
||||
print(f" BAD {f}: missing or zero bytes" )
|
||||
bad += 1
|
||||
else:
|
||||
total += os.path.getsize(p)
|
||||
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
|
||||
raise SystemExit(1 if bad else 0)
|
||||
PY
|
||||
|
||||
# Shard count gate (what the serve script and deployer check).
|
||||
n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
||||
if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
|
||||
echo " BAD shard count: found $n, want $EXPECTED_SHARDS" >&2
|
||||
fail=1
|
||||
else
|
||||
echo " OK shard count $n"
|
||||
fi
|
||||
|
||||
if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
|
||||
echo "hashing every shard (slow)..."
|
||||
cd "$DEST"
|
||||
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
|
||||
sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand
|
||||
fi
|
||||
|
||||
[[ "$fail" -eq 0 ]] || exit 1
|
||||
echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
|
||||
Reference in new issue
Block a user