feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain

Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq ->
orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7
shards, its own single-layer MTP head). The quantization-aware entrypoint
serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP
self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8
(66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x),
battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/
248.9). MXFP4 checkpoint stays on disk as the rollback target.

Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256>
manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte +
sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate
manifest -> download -> detect quantization -> edit the quadlet -> deploy).

Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes,
AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
Frank committed 2026-09-20 21:53:00 -07:00
1 parent 2458d754ce
commit c1a4ccbbcd
8 files changed
+534 -45

No files matched your search

@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium vllm-radiance MXFP4 Runbook
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era.
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
tags:
- frank-medium
- dankserver
@@ -28,6 +28,16 @@ It replaced the mainline-llama.cpp layer-split deployment
model had earlier run here as two single-GPU replicas, which is the shape most
of this page describes.
**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
shards) with MTP self-speculation — the quantization-aware entrypoint reads
`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
checkpoint above remains on disk as the rollback target. The swap process,
the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
the "Deployed flags" table below still shows the MXFP4 shape (the rollback
target).
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
@@ -43,8 +53,8 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Setting | Value |
|---|---|
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) |
| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) |
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
@@ -56,17 +66,17 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
| Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |
**The KV pool is auto-sized, not pinned.** The arm-R benchmark run used
`KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token
sequence (262,720 tokens at 70,656 B/token — a profile inflated by the
8192-chunk activation peak). Production ships no such pin: the pool came up at
**943,581 tokens = 3.59× the full window**, and the boot line read
`Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`.
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8
checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it
leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540
tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`,
`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4
rollback target auto-sized to 943,581 tokens (3.59×) on the same cards.
Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
full-attention layers plus the drafter's. Container RSS measured 28.4 GiB
against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
two-replica vision shape, not this one.
full-attention layers plus the MTP drafter's. After the first healthy start,
recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
cards.
## Prefix caching — measured granularity (2026-09-20)