feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain
Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
@@ -58,13 +58,20 @@ frank-small, port 7771]
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
|
||||
2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
|
||||
drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
|
||||
battery that chose it, the auto-sized 943,581-token KV pool, verified
|
||||
capabilities (tools, reasoning, vision), and the record of the 2026-09-16
|
||||
two-replica era
|
||||
2026-09-20: vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
|
||||
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
|
||||
shards) as ONE tensor-parallel process (TP=2) behind :7770, with MTP
|
||||
self-speculation (the model's own single-layer MTP head). The quantization-
|
||||
aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint
|
||||
(just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8
|
||||
drafter) remains on disk as the rollback target. The three-arm battery that
|
||||
chose TP=2, the auto-sized KV pool, and verified capabilities (tools,
|
||||
reasoning, vision)
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||||
|
||||
Reference in New Issue
Block a user