dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0

Browse Source 
- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards,
  quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on)
- Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel,
  built on the published 0.9.3 base; docker->podman via save/load)
- GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16)
- Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile +
  sitecustomize registration); GPU_UTIL default for set -u robustness
- Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir
- deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults
- Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback
- Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6)
- Runbook: MXFP6 active shape, MXFP4 rollback

MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
Frank
2026-09-21 22:46:44 -07:00
parent 600812d459
commit e9a01cd013
6 changed files with 709 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+40 -42
View File
@@ -1,48 +1,47 @@
---
type: configuration
title: frank-medium vllm-radiance MXFP4 Runbook
description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
title: frank-medium vllm-radiance MXFP6 Runbook
description: ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Cut to MXFP6 on 2026-09-21 (PR #51's launch6_p kernel, image 0.13.0-e060895); the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the 2026-09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
tags:
- frank-medium
- dankserver
- vllm
- vllm-radiance
- mxfp6
- mxfp4
- gfx1201
- r9700
- rccl
- pcie-atomics
- quadlet
timestamp: 2026-09-20
timestamp: 2026-09-21
---
# frank-medium vllm-radiance MXFP4 Runbook
# frank-medium vllm-radiance MXFP6 Runbook
The ACTIVE frank-medium deployment: the `stilldeadcode/vllm-radiance:0.9.3`
image (tagged locally as `localhost/radiance-vllm:0.9.3`) plus the
`ggz14/radiance-vllm-mxfp4` patches, serving
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` as **ONE
tensor-parallel process across both Radeon AI PRO R9700 (gfx1201)**.
It replaced the mainline-llama.cpp layer-split deployment
([[frank-medium-llamacpp-dflash2]], SUPERSEDED 2026-09-20); the same image and
model had earlier run here as two single-GPU replicas, which is the shape most
of this page describes.
The ACTIVE frank-medium deployment: the `ggz14/vllm-radiance-mxfp4:0.13.0-e060895`
image (built from PR #51's head, tagged locally as
`localhost/radiance-vllm:0.13.0-e060895`) plus the `radiance-vllm-mxfp4`
patches at e060895, serving `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6`
(ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards) as **ONE tensor-parallel
process across both Radeon AI PRO R9700 (gfx1201)**. It replaced the
mainline-llama.cpp layer-split deployment ([[frank-medium-llamacpp-dflash2]],
SUPERSEDED 2026-09-20); the same image and model had earlier run here as two
single-GPU replicas, which is the shape most of this page describes.
**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This
endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3,
`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has
since been cut back to the MXFP4 checkpoint above, now with
`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's
historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the
flag was on the common exec path), so it could not explain the FP8-vs-MXFP4
*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top
compounds attention-path precision loss, which shows up as degraded *output
quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8
banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per
request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the
full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is
the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with
`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window).
**Serving shape since 2026-09-21: ParoQuant MXFP6 with unquantized (bf16) KV.**
This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel,
`quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on,
e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and
`--gpu-memory-utilization 0.99`. The Swift variant is a finetune of
`ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the
DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may
drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec
decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to
`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, `GPU_UTIL` to 0.98, and the image
back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/
vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to
486,464 tokens with bf16 KV and held the full 262,144 window).
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
@@ -59,31 +58,30 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Setting | Value |
|---|---|
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) |
| Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~730k tokens** (estimated at 0.99 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~19.2 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` |
| Prompt cache | `--enable-prefix-caching` |
| GPU fill | `--gpu-memory-utilization 0.98` |
| GPU fill | `--gpu-memory-utilization 0.99` |
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
| Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) |
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
| Chat template | `qwen-sharp-v22.4.0.jinja` (froggeric Qwen-Sharp v22.4.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/patches/qwen-fixed-v22.3.jinja`) |
| Chat template | `qwen-sharp-v22.5.0.jinja` (peculiar-ragdoll Qwen-Sharp v22.5.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/templates/qwen-sharp-v22.4.0.jinja`) |
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for
the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21
clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464
tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the
full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8
KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to
384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which
is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that
affords clean KV at the native 262,144. After the first healthy start,
recalibrate with `calibrate-kv.sh`.
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2), which at 0.99 fill
(34.2 GiB per card) leaves ~19.2 GiB per card for KV + activations. The
estimated pool is **~730k tokens** (the actual size is determined at boot; the
full 262,144 window fits with ~2.7x headroom). The DFlash2-FP8 drafter costs
2.1 GiB of VRAM on top. For reference the MXFP4 checkpoint (now the rollback)
auto-sized to 486,464 tokens with bf16 KV (1.86x the window), and the FP8
checkpoint would only reach ~192,000 unquantized — which is why MXFP6's larger
weights still afford clean KV at the native 262,144 (the 0.99 fill buys the
pool back). After the first healthy start, recalibrate with `calibrate-kv.sh`.
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)