feat(frank-medium): swap MXFP4 -> FP8; add the one-command swap toolchain
Checkpoint: just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq -> orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 30.9 GB / 7 shards, its own single-layer MTP head). The quantization-aware entrypoint serves the plain FP8 path: --quantization fp8, ROCM_AITER_UNIFIED_ATTN, MTP self-speculation (SPEC_METHOD=mtp, SPEC=8). Measured: MTP mean accept 5.66/8 (66.2% draft acceptance), KV pool auto-sized to 461,540 tokens (1.76x), battery 85.9 t/s decode / 0.06 s TTFT / 179.7 burst-4 (vs MXFP4 92.4/0.22/ 248.9). MXFP4 checkpoint stays on disk as the rollback target. Streamlining: scripts/hf-manifest.py (generate a <filename> <size> <sha256> manifest from the live repo tree), scripts/hf-fetch.sh (resumable, byte + sha256 verified, idempotent fetch), scripts/swap-frank-medium.sh (generate manifest -> download -> detect quantization -> edit the quadlet -> deploy). Docs: new frank-medium-model-swap runbook; radiance runbook, both indexes, AGENTS.md, and log.md flipped to the FP8 shape.
This commit is contained in:
1 parent
2458d754ce
commit
c1a4ccbbcd
8 files changed
+534
-45
No files matched your search
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-medium vllm-radiance MXFP4 Runbook
|
||||
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era.
|
||||
description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance serving orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, MTP self-speculation) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). The quantization-aware entrypoint serves the plain FP8 path; the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq + the z-lab DFlash2-FP8 drafter) remains on disk as the rollback target. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
@@ -28,6 +28,16 @@ It replaced the mainline-llama.cpp layer-split deployment
|
||||
model had earlier run here as two single-GPU replicas, which is the shape most
|
||||
of this page describes.
|
||||
|
||||
**Checkpoint swapped to FP8 on 2026-09-20.** The endpoint now serves
|
||||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise, 30.9 GB / 7
|
||||
shards) with MTP self-speculation — the quantization-aware entrypoint reads
|
||||
`quant_method` from `config.json` and serves the plain FP8 path (no MXFP4
|
||||
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`). The MXFP4
|
||||
checkpoint above remains on disk as the rollback target. The swap process,
|
||||
the toolchain, and the measured trade-off are in [[frank-medium-model-swap]];
|
||||
the "Deployed flags" table below still shows the MXFP4 shape (the rollback
|
||||
target).
|
||||
|
||||
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
|
||||
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
|
||||
ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
|
||||
@@ -43,8 +53,8 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` |
|
||||
| Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) |
|
||||
| Model | `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128, 30.9 GB / 7 shards) |
|
||||
| Speculative | `--speculative-config {"method":"mtp","num_speculative_tokens":8}` (the model's own single-layer MTP head; mean acceptance 5.66/8, 66.2% draft acceptance) |
|
||||
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
|
||||
| Context | `--max-model-len 262144` |
|
||||
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
|
||||
@@ -56,17 +66,17 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
|
||||
| Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |
|
||||
|
||||
**The KV pool is auto-sized, not pinned.** The arm-R benchmark run used
|
||||
`KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token
|
||||
sequence (262,720 tokens at 70,656 B/token — a profile inflated by the
|
||||
8192-chunk activation peak). Production ships no such pin: the pool came up at
|
||||
**943,581 tokens = 3.59× the full window**, and the boot line read
|
||||
`Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`.
|
||||
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
|
||||
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights leave. The FP8
|
||||
checkpoint is bigger than the MXFP4 one (30.9 vs 19.4 GiB of weights), so it
|
||||
leaves less for the pool: the 2026-09-20 FP8 start auto-sized to **461,540
|
||||
tokens = 1.76× the full window** (`GPU KV cache size: 461,540 tokens`,
|
||||
`Maximum concurrency for 262,144 tokens per request: 1.76x`). The MXFP4
|
||||
rollback target auto-sized to 943,581 tokens (3.59×) on the same cards.
|
||||
Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
|
||||
full-attention layers plus the drafter's. Container RSS measured 28.4 GiB
|
||||
against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
|
||||
max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
|
||||
two-replica vision shape, not this one.
|
||||
full-attention layers plus the MTP drafter's. After the first healthy start,
|
||||
recalibrate with `calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB
|
||||
cards.
|
||||
|
||||
## Prefix caching — measured granularity (2026-09-20)
|
||||
|
||||
|
||||
Reference in new issue
Block a user