frank-primary: roll back to qwen FP8 + MTP4 (2339fae trio)
DFlash2 experiment declared failed 2026-10-01: the drafter serves and drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL text classes including verbatim repetition — the aux-hidden-state capture semantics are wrong (single-stream entering-state vs post-layer/comb is the untested alternative). At ~1.2 acceptance the 8-token verify tax makes decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured). Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4, prefetch off, 200k x 3 seqs, fp8 KV.
This commit is contained in:
1 parent
caee26e3c3
commit
d91379cff2
5 files changed
+88
-199
No files matched your search
@@ -1,24 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# Payload inventory/verification for frank-primary:
|
||||
# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128,
|
||||
# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv /
|
||||
# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the
|
||||
# int4 payload that became frank-primary on 2026-09-30.
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
|
||||
# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
|
||||
# became frank-primary on 2026-09-29.
|
||||
#
|
||||
# bash scripts/download-frank-primary.sh --verify-only (the default)
|
||||
# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode)
|
||||
# bash scripts/download-frank-primary.sh --fetch
|
||||
#
|
||||
# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at
|
||||
# bring-up). The LFS oids are still not verified byte-exactly here — the tree
|
||||
# API masks them — so integrity rests on the weight-map index, the shard count
|
||||
# and the loader's own tensor-shape checks at boot.
|
||||
#
|
||||
# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which
|
||||
# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod
|
||||
# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A
|
||||
# compressed-tensors W4A16 checkpoint of the same model (e.g.
|
||||
# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which
|
||||
# has no offload recipe, and refuses to serve at all on this host.
|
||||
# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
|
||||
# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
|
||||
# manifest cannot be built without the authed endpoint. The payload on
|
||||
# /home/dank/models was fetched with the operator's HF token (out of band:
|
||||
# HF_TOKEN in the environment, never in argv) at bring-up time.
|
||||
#
|
||||
# Modes:
|
||||
# --fetch resume any missing/truncated shard via `hf download` with
|
||||
@@ -31,17 +24,16 @@
|
||||
# (slow, ~10 min for 173 GiB).
|
||||
set -euo pipefail
|
||||
|
||||
REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound"
|
||||
DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound"
|
||||
EXPECTED_SHARDS=34
|
||||
REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
|
||||
EXPECTED_SHARDS=131
|
||||
|
||||
mode="verify"
|
||||
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||
|
||||
if [[ "$mode" == "fetch" ]]; then
|
||||
# The repo is public, so no token is required; if one is present it is taken
|
||||
# from the environment and never from argv (a -H header is readable in
|
||||
# /proc/<pid>/cmdline — the 2026-09-16 leak).
|
||||
: "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
|
||||
export HF_TOKEN
|
||||
mkdir -p "$DEST"
|
||||
hf download "$REPO" --local-dir "$DEST"
|
||||
fi
|
||||
@@ -89,7 +81,7 @@ if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
|
||||
echo "hashing every shard (slow)..."
|
||||
cd "$DEST"
|
||||
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
|
||||
sha256sum model-*-of-00034.safetensors > /dev/null # presence check; content hash list printed on demand
|
||||
sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand
|
||||
fi
|
||||
|
||||
[[ "$fail" -eq 0 ]] || exit 1
|
||||
|
||||
Reference in new issue
Block a user