feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain
Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
1 parent
2458d754ce
commit
c1a4ccbbcd
8 files changed
+534
-45
No files matched your search
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
|
||||
|
||||
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||||
19.37 GiB single shard, DFlash2 external drafter) →
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128,
|
||||
dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head).
|
||||
The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads
|
||||
`quant_method` from `config.json` and serves the plain FP8 path — no MXFP4
|
||||
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP
|
||||
self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four
|
||||
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler
|
||||
(`deploy-remote.sh`) now reads the serving shape from the quadlet and gates
|
||||
on the right assets (drafter only for `dflash`, `r4d.so` only for
|
||||
`mxfp4`/`quark`).
|
||||
- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean
|
||||
acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates
|
||||
0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22
|
||||
tensors), so self-speculation needs no external drafter and no `fp8_mtp.py`
|
||||
rewrite.
|
||||
- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than
|
||||
MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4
|
||||
GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with
|
||||
`calibrate-kv.sh` after the first healthy start.
|
||||
- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %),
|
||||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||||
(−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the
|
||||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %,
|
||||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||||
vs R4D). Vision, tools, and reasoning all verified live.
|
||||
- **Streamlining — the long pole was the download, not the GPU.** Three tools
|
||||
make a swap one command: `scripts/hf-manifest.py` (generate a
|
||||
`<filename> <size> <sha256>` manifest from the live repo tree — LFS sha256s
|
||||
from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`),
|
||||
`scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token
|
||||
out-of-band via a mode-600 `curl -K` file), and
|
||||
`scripts/swap-frank-medium.sh` (generate manifest → download → detect
|
||||
quantization → edit the quadlet → deploy). A swap is now
|
||||
`swap-frank-medium.sh --repo <owner/repo> --target /home/dank/models/<name>
|
||||
--generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]].
|
||||
- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table
|
||||
+ KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the
|
||||
rollback target); [[frank-medium-model-swap]] is the new swap runbook; the
|
||||
LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk
|
||||
at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`.
|
||||
|
||||
## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
|
||||
|
||||
- **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`
|
||||
|
||||
Reference in new issue
Block a user