feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain

Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq ->
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards, its own single-layer MTP head). The quantization-aware entrypoint
serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP
self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8
(66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x),
battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/
248.9). MXFP4 checkpoint stays on disk as the rollback target.

Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256>
manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte +
sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate
manifest -> download -> detect quantization -> edit the quadlet -> deploy).

Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes,
AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank committed 2026-09-20 21:53:00 -07:00
1 parent 2458d754ce
commit c1a4ccbbcd
8 files changed
+534 -45

No files matched your search

+45
View File
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
# Bundle Change Log
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
19.37 GiB single shard, DFlash2 external drafter) →
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128,
dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head).
The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads
`quant_method` from `config.json` and serves the plain FP8 path — no MXFP4
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP
self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler
(`deploy-remote.sh`) now reads the serving shape from the quadlet and gates
on the right assets (drafter only for `dflash`, `r4d.so` only for
`mxfp4`/`quark`).
- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean
acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates
0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22
tensors), so self-speculation needs no external drafter and no `fp8_mtp.py`
rewrite.
- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than
MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4
GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with
`calibrate-kv.sh` after the first healthy start.
- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %),
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
(−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the
expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %,
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
vs R4D). Vision, tools, and reasoning all verified live.
- **Streamlining — the long pole was the download, not the GPU.** Three tools
make a swap one command: `scripts/hf-manifest.py` (generate a
`<filename> <size> <sha256>` manifest from the live repo tree — LFS sha256s
from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`),
`scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token
out-of-band via a mode-600 `curl -K` file), and
`scripts/swap-frank-medium.sh` (generate manifest → download → detect
quantization → edit the quadlet → deploy). A swap is now
`swap-frank-medium.sh --repo <owner/repo> --target /home/dank/models/<name>
--generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]].
- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table
+ KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the
rollback target); [[frank-medium-model-swap]] is the new swap runbook; the
LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk
at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`.
## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
- **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`