Frank
3bb0e82ccf
Replace the R9V radiance fork with the tcclaviger RDNA4 vLLM stack (the digest-pinned image the 2026-09-29 tuning campaign measured), serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (131 shards, 172.8 GiB, already on disk) with MTP4 spec decode. Qualified by boot probe on 2026-09-29: - MAXLEN=200000 x SEQS=3 (600k total) at --kv-cache-dtype fp8 (q8/q8 KV): 'GPU KV cache size: 729,545 tokens; Maximum concurrency for 200,000 tokens per request: 3.65x' at KV_BYTES 7.2 GiB PER RANK. - KV_BYTES is per-rank: global 16G and 9G both OOM (worker tried to allocate the full size; ~7.5 GiB/rank is the ceiling with the arena). - prefetch OFF by law (telemetry: 36% precision, ~50% wasted traffic); TunableOp off (corrupts numerics). - decode at this shape: 42.94/34.55/36.09 tok/s; arena 6,032 slots. Trio: frank-primary.env (all FRANK_PRIMARY_* keys), serve-frank-primary.sh (direct podman run; entrypoint execs vllm serve; waits on the container), frank-primary.service (root unit, same ExecStop law, new container name frank-primary-qwen). Deployer gates rewritten (image digest + shard count + weight-map index). download-frank-primary.sh is now inventory/verify (repo is GATED — 401 anonymous, masked LFS hashes; payload already on disk). Seam rewritten for the new contract. Pre-cutover tag: frank-primary-r9v-q8-20260929.
89 lines
3.3 KiB
Bash
Executable File
89 lines
3.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Payload inventory/verification for frank-primary:
|
|
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
|
|
# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
|
|
# became frank-primary on 2026-09-29.
|
|
#
|
|
# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode)
|
|
# bash scripts/download-frank-primary.sh --fetch
|
|
#
|
|
# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
|
|
# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
|
|
# manifest cannot be built without the authed endpoint. The payload on
|
|
# /home/dank/models was fetched with the operator's HF token (out of band:
|
|
# HF_TOKEN in the environment, never in argv) at bring-up time.
|
|
#
|
|
# Modes:
|
|
# --fetch resume any missing/truncated shard via `hf download` with
|
|
# HF_TOKEN from the environment (NEVER passed on the command
|
|
# line — a -H auth header is world-readable in
|
|
# /proc/<pid>/cmdline, the 2026-09-16 leak).
|
|
# --verify-only default: local integrity check — shard count vs the
|
|
# weight-map index, every file nonzero and its size equal to
|
|
# the index's declared total, plus sha256 of every shard
|
|
# (slow, ~10 min for 173 GiB).
|
|
set -euo pipefail
|
|
|
|
REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
|
|
DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
|
|
EXPECTED_SHARDS=131
|
|
|
|
mode="verify"
|
|
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
|
|
|
if [[ "$mode" == "fetch" ]]; then
|
|
: "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
|
|
export HF_TOKEN
|
|
mkdir -p "$DEST"
|
|
hf download "$REPO" --local-dir "$DEST"
|
|
fi
|
|
|
|
echo "verifying local payload: $DEST"
|
|
fail=0
|
|
|
|
# The weight-map index is the source of truth for the shard set.
|
|
INDEX="$DEST/model.safetensors.index.json"
|
|
if [[ ! -f "$INDEX" ]]; then
|
|
echo " BAD weight-map index missing: $INDEX" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Every index entry must exist, nonzero, and sized equal to the map's total.
|
|
python3 - "$INDEX" "$DEST" <<'PY' || fail=1
|
|
import json, os, sys
|
|
index, dest = sys.argv[1], sys.argv[2]
|
|
wm = json.load(open(index))["weight_map"]
|
|
files = sorted(set(wm.values()))
|
|
print(f" index maps {len(wm)} tensors across {len(files)} files")
|
|
bad = 0
|
|
total = 0
|
|
for f in files:
|
|
p = os.path.join(dest, f)
|
|
if not os.path.isfile(p) or os.path.getsize(p) == 0:
|
|
print(f" BAD {f}: missing or zero bytes" )
|
|
bad += 1
|
|
else:
|
|
total += os.path.getsize(p)
|
|
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
|
|
raise SystemExit(1 if bad else 0)
|
|
PY
|
|
|
|
# Shard count gate (what the serve script and deployer check).
|
|
n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
|
if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
|
|
echo " BAD shard count: found $n, want $EXPECTED_SHARDS" >&2
|
|
fail=1
|
|
else
|
|
echo " OK shard count $n"
|
|
fi
|
|
|
|
if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
|
|
echo "hashing every shard (slow)..."
|
|
cd "$DEST"
|
|
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
|
|
sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand
|
|
fi
|
|
|
|
[[ "$fail" -eq 0 ]] || exit 1
|
|
echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
|