feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain

Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq ->
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards, its own single-layer MTP head). The quantization-aware entrypoint
serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP
self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8
(66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x),
battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/
248.9). MXFP4 checkpoint stays on disk as the rollback target.

Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256>
manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte +
sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate
manifest -> download -> detect quantization -> edit the quadlet -> deploy).

Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes,
AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank committed 2026-09-20 21:53:00 -07:00
1 parent 2458d754ce
commit c1a4ccbbcd
8 files changed
+534 -45

No files matched your search

+10 -5
View File
@@ -116,11 +116,16 @@ The LLM inference fleet on the two hosts. Full index:
- [[deployment]] — unified deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the
R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq
with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700
behind the :7770 endpoint — deployed flags, the three-arm battery that chose
it, verified capabilities, and the record of the 2026-09-16 two-replica era
2026-09-20: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D
one-shot allreduce) serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3
block-wise, MTP self-speculation) on Dankserver's 2x Radeon AI PRO R9700
behind the :7770 endpoint — the quantization-aware entrypoint, the
three-arm battery that chose TP=2, verified capabilities, and the record of
the 2026-09-16 two-replica era
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
quadlet edit, the deploy, the capability battery, and the rollback path
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two