frank-primary: roll back to qwen FP8 + MTP4 (2339fae trio)

DFlash2 experiment declared failed 2026-10-01: the drafter serves and
drafts (12.8 tok/s proposed) but acceptance is pinned at 1.0-1.38 on ALL
text classes including verbatim repetition — the aux-hidden-state capture
semantics are wrong (single-stream entering-state vs post-layer/comb is
the untested alternative). At ~1.2 acceptance the 8-token verify tax makes
decode 5x SLOWER than the 9.9-10.0 baseline (1.6-1.8 tok/s measured).

Restores the verified 37-40 tok/s shape: qwen3.8-flash-next FP8, MTP4,
prefetch off, 200k x 3 seqs, fp8 KV.
This commit is contained in:
Frank committed 2026-09-30 20:13:23 -07:00
1 parent caee26e3c3
commit d91379cff2
5 files changed
+88 -199

No files matched your search

+15 -23
View File
@@ -1,24 +1,17 @@
#!/usr/bin/env bash
# Payload inventory/verification for frank-primary:
# Intel/GLM-5.3-Flash-W4A16-AutoRound (AutoRound symmetric int4, group 128,
# packing "auto_round:auto_gptq": 34 numbered shards + model_extra_conv /
# model_extra_tensors, weight-map index, 113,074 tensors, ~170 GiB) — the
# int4 payload that became frank-primary on 2026-09-30.
# orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (compressed-tensors FP8, 131
# shards + weight-map index, ~172.8 GiB) — the tcclaviger-stack payload that
# became frank-primary on 2026-09-29.
#
# bash scripts/download-frank-primary.sh --verify-only (the default)
# bash scripts/download-frank-primary.sh --verify-only (the ONLY mode)
# bash scripts/download-frank-primary.sh --fetch
#
# THIS REPO IS PUBLIC (measured 2026-09-30: the payload fetched anonymously at
# bring-up). The LFS oids are still not verified byte-exactly here — the tree
# API masks them — so integrity rests on the weight-map index, the shard count
# and the loader's own tensor-shape checks at boot.
#
# WHY THIS QUANT: its quantization_config.quant_method is "auto-round", which
# the fork overrides to "inc" — that is the ONLY route to R4dW4a16MoEMethod
# (libr4d's grouped W4A16 GEMM) and therefore to an expert-offload recipe. A
# compressed-tensors W4A16 checkpoint of the same model (e.g.
# dealignai/...-CYBERSECURITY-W4A16) reaches TritonWNA16Experts instead, which
# has no offload recipe, and refuses to serve at all on this host.
# THIS REPO IS GATED (401 anonymous — measured 2026-09-29): anonymous fetch
# is impossible and the tree API masks LFS hashes, so a byte-exact upstream
# manifest cannot be built without the authed endpoint. The payload on
# /home/dank/models was fetched with the operator's HF token (out of band:
# HF_TOKEN in the environment, never in argv) at bring-up time.
#
# Modes:
# --fetch resume any missing/truncated shard via `hf download` with
@@ -31,17 +24,16 @@
# (slow, ~10 min for 173 GiB).
set -euo pipefail
REPO="Intel/GLM-5.3-Flash-W4A16-AutoRound"
DEST="/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound"
EXPECTED_SHARDS=34
REPO="orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8"
DEST="/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8"
EXPECTED_SHARDS=131
mode="verify"
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
if [[ "$mode" == "fetch" ]]; then
# The repo is public, so no token is required; if one is present it is taken
# from the environment and never from argv (a -H header is readable in
# /proc/<pid>/cmdline — the 2026-09-16 leak).
: "${HF_TOKEN:?--fetch needs HF_TOKEN in the environment (never argv)}"
export HF_TOKEN
mkdir -p "$DEST"
hf download "$REPO" --local-dir "$DEST"
fi
@@ -89,7 +81,7 @@ if [[ "${2:-}" == "--hash" || "${VERIFY_HASH:-0}" == "1" ]]; then
echo "hashing every shard (slow)..."
cd "$DEST"
sha256sum -c --quiet model.safetensors.index.json 2>/dev/null || \
sha256sum model-*-of-00034.safetensors > /dev/null # presence check; content hash list printed on demand
sha256sum model-*-of-00131.safetensors > /dev/null # presence check; content hash list printed on demand
fi
[[ "$fail" -eq 0 ]] || exit 1