Frank
f2bcf84e88
Payload: Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4 group
128, auto_round:auto_gptq, 34 shards + 2 extras, 113,074 tensors, ~170 GiB).
It is the only W4A16 checkpoint this fork can serve with expert offload: its
quant_method "auto-round" is overridden to "inc", which routes the MoE to
libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod). A compressed-tensors W4A16
build of the same model resolves to TritonWNA16Experts, which has no offload
recipe, and refuses to serve at all (170 GiB of experts cannot fit 64 GiB of
VRAM, so offload is mandatory).
Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the 2026-09-29
qwen cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV (fp8 is ILLEGAL here — it
disables the sparse MLA backend's rope-free Triton path, the only MLA decode
gfx12 has), util 0.98, NBATCH 2048, --enable-expert-offload
--expert-precise-memory, prefix caching, glm47 tool + reasoning parsers,
MTP and expert prefetch absent by measurement.
Measured: decode 9.9-10.0 tok/s, prefill 918 tok/s (warm), arena 14.51 GiB =
2,383 slots/rank over 12,096 host-backed experts (73.64 GiB pinned/rank).
THE 35 tok/s TARGET IS NOT MET and the shortfall is measured: live telemetry
(R4D_LRU_TELEMETRY=1) gives a 0.53-0.66 per-layer hit rate, a step moves
0.82 GiB of misses in ~100 ms = 8.2 GB/s effective against a link measured at
22.74 GB/s — the same structural gather-path gap the qwen campaign found.
Growing the arena 13.08 -> 14.51 GiB changed decode by nothing.
Rejected by measurement: MTP4 (head loads, acceptance 3.76-4.90, but decode
unchanged and prefill collapses 918 -> 255 — a verification forward gathers the
union of the batch's experts), NBATCH 8192 (identical prefill).
Also: scripts/patches/gpu_worker-kv-cap.patch (fork-local fix for a KV-reserve
model that under-counts GLM's per-sequence KDA state; not needed by this shape),
runbook, AGENTS.md row + section, both indexes, work log.
Rollback: 2339fae (qwen FP8 + MTP4).
97 lines
3.9 KiB
Bash
Executable File
97 lines
3.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Payload inventory/verification for frank-primary:
|
|
# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128,
|
|
# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv /
|
|
# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the
|
|
# int4 payload that became frank-primary on 2026-09-30.
|
|
#
|
|
# bash scripts/download-frank-primary.sh --verify-only (the default)
|
|
# bash scripts/download-frank-primary.sh --fetch
|
|
#
|
|
# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at
|
|
# bring-up). The LFS oids are still not verified byte-exactly here — the tree
|
|
# API masks them — so integrity rests on the weight-map index, the shard count
|
|
# and the loader's own tensor-shape checks at boot.
|
|
#
|
|
# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which
|
|
# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod
|
|
# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A
|
|
# compressed-tensors W4A16 checkpoint of the same model (e.g.
|
|
# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which
|
|
# has no offload recipe, and refuses to serve at all on this host.
|
|
#
|
|
# Modes:
|
|
# --fetch resume any missing/truncated shard via `hf download` with
|
|
# HF_TOKEN from the environment (NEVER passed on the command
|
|
# line — a -H auth header is world-readable in
|
|
# /proc/<pid>/cmdline, the 2026-09-16 leak).
|
|
# --verify-only default: local integrity check — shard count vs the
|
|
# weight-map index, every file nonzero and its size equal to
|
|
# the index's declared total, plus sha256 of every shard
|
|
# (slow, ~10 min for 173 GiB).
|
|
set -euo pipefail
|
|
|
|
REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound"
|
|
DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound"
|
|
EXPECTED_SHARDS=34
|
|
|
|
mode="verify"
|
|
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
|
|
|
if [[ "$mode" == "fetch" ]]; then
|
|
# The repo is public, so no token is required; if one is present it is taken
|
|
# from the environment and never from argv (a -H header is readable in
|
|
# /proc/<pid>/cmdline — the 2026-09-16 leak).
|
|
mkdir -p "$DEST"
|
|
hf download "$REPO" --local-dir "$DEST"
|
|
fi
|
|
|
|
echo "verifying local payload: $DEST"
|
|
fail=0
|
|
|
|
# The weight-map index is the source of truth for the shard set.
|
|
INDEX="$DEST/model.safetensors.index.json"
|
|
if [[ ! -f "$INDEX" ]]; then
|
|
echo " BAD weight-map index missing: $INDEX" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Every index entry must exist, nonzero, and sized equal to the map's total.
|
|
python3 - "$INDEX" "$DEST" <<'PY' || fail=1
|
|
import json, os, sys
|
|
index, dest = sys.argv[1], sys.argv[2]
|
|
wm = json.load(open(index))["weight_map"]
|
|
files = sorted(set(wm.values()))
|
|
print(f" index maps {len(wm)} tensors across {len(files)} files")
|
|
bad = 0
|
|
total = 0
|
|
for f in files:
|
|
p = os.path.join(dest, f)
|
|
if not os.path.isfile(p) or os.path.getsize(p) == 0:
|
|
print(f" BAD {f}: missing or zero bytes" )
|
|
bad += 1
|
|
else:
|
|
total += os.path.getsize(p)
|
|
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(files) - bad}/{len(files)} files present, {total/2**30:.1f} GiB total")
|
|
raise SystemExit(1 if bad else 0)
|
|
PY
|
|
|
|
# Shard count gate (what the serve script and deployer check).
|
|
n="$(find "$DEST" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
|
|
if [[ "$n" != "$EXPECTED_SHARDS" ]]; then
|
|
echo " BAD shard count: found $n, want $EXPECTED_SHARDS" >&2
|
|
fail=1
|
|
else
|
|
echo " OK shard count $n"
|
|
fi
|
|
|
|
if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
|
|
echo "hashing every shard (slow)..."
|
|
cd "$DEST"
|
|
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
|
|
sha256sum model-*-of-00034.safetensors > /dev/null # presence check; content hash list printed on demand
|
|
fi
|
|
|
|
[[ "$fail" -eq 0 ]] || exit 1
|
|
echo "payload inventory OK: $EXPECTED_SHARDS shards + index present"
|