frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0
- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards, quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on) - Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel, built on the published 0.9.3 base; docker->podman via save/load) - GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16) - Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile + sitecustomize registration); GPU_UTIL default for set -u robustness - Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir - deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults - Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback - Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6) - Runbook: MXFP6 active shape, MXFP4 rollback MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
Frank committed
1 parent
600812d459
commit
e9a01cd013
6 files changed
+709
-111
No files matched your search
@@ -4,18 +4,18 @@
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium): the vllm-radiance
|
||||
# stack (stilldeadcode/vllm-radiance:0.9.3 image) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 / quark, W4A8, 19.37
|
||||
# GiB) on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process
|
||||
# stack (the 0.13.0-e060895 image, PR #51) serving
|
||||
# hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, W4A8, 25.16 GiB)
|
||||
# on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process
|
||||
# (TP=2, R4D one-shot all-reduce) with the z-lab DFlash2 external drafter
|
||||
# (SPEC=7) and UNQUANTIZED bf16 KV (KV_DTYPE=auto).
|
||||
# The entrypoint is QUANTIZATION-AWARE: it reads the model's weight
|
||||
# quantization from config.json and serves the plain FP8 path (no MXFP4
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4
|
||||
# patch-and-serve path (the DFlash2 drafter). A model swap is now: change
|
||||
# SNAP (+ SPEC_METHOD/SPEC), restart. The FP8 checkpoint
|
||||
# (orcarouter/Qwen3.8-27B-Uncensored-FP8) is the on-disk rollback target — the
|
||||
# 2026-09-20 to 2026-09-21 serving shape, before this MXFP4 + clean-KV cut.
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention), the MXFP4
|
||||
# patch-and-serve path, or the ParoQuant MXFP6 stack (PR #51's launch6_p
|
||||
# kernel). A model swap is now: change SNAP (+ SPEC_METHOD/SPEC), restart.
|
||||
# The MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the
|
||||
# 2026-09-20 to 2026-09-21 serving shape) is the on-disk rollback target.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
@@ -58,19 +58,19 @@
|
||||
# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
|
||||
# the registry.
|
||||
#
|
||||
# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
|
||||
# KEEP IN SYNC: the radiance clone is pinned (e060895...); if it is updated,
|
||||
# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
|
||||
# serve-mxfp4.sh container command and bump the pin deliberately.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
Description=frank-medium — vllm-radiance ParoQuant MXFP6 Swift Qwen3.8-27B on 2x R9700 (gfx1201)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
Image=localhost/radiance-vllm:0.13.0-e060895
|
||||
# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
@@ -85,14 +85,14 @@ Environment=HIP_VISIBLE_DEVICES=0,1
|
||||
# Drafting: the external z-lab DFlash2-FP8 drafter (SPEC_METHOD=dflash). The
|
||||
# plain-FP8 checkpoint (rollback) carries its own single-layer MTP head, but
|
||||
# DFlash2 beat MTP self-speculation on decode, so the drafter serves either.
|
||||
# Chat template: the Qwen-Sharp variant (froggeric v22.4.0) — the terse
|
||||
# Chat template: the Qwen-Sharp variant (peculiar-ragdoll v22.5.0) — the terse
|
||||
# system prompt (opt out per-request with chat_template_kwargs {"terse":
|
||||
# false}), message.reasoning support (vLLM's thinking field), and the
|
||||
# tightened tool-call rendering. The predecessor (qwen-fixed-v22.3.jinja in
|
||||
# the /patches clone) remains there as the rollback target: flip CHAT_TEMPLATE.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
# tightened tool-call rendering. The predecessor (qwen-sharp-v22.4.0.jinja,
|
||||
# mounted beside it) remains as the rollback target: flip CHAT_TEMPLATE.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-PARO-MXFP6
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.4.0.jinja
|
||||
Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.5.0.jinja
|
||||
# Fleet identity: the id at /v1/models and in every response body. Every
|
||||
# client (Hermes, the harness, services.json) points at frank-medium.
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
@@ -116,7 +116,7 @@ Environment=MAXLEN=262144
|
||||
# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
|
||||
# image input on either checkpoint. harness models.yml keeps frank-medium
|
||||
# input: [text, image].
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=GPU_UTIL=0.99
|
||||
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
|
||||
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2
|
||||
# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4)
|
||||
@@ -144,7 +144,7 @@ Volume=/home/dank/models:/models
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Keyed on model + stack version + every graph-changing knob;
|
||||
# never share it across configurations (upstream rule).
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
Volume=/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
|
||||
@@ -154,6 +154,7 @@ Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.4.0.jinja:/templates/qwen-sharp-v22.4.0.jinja:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja:/templates/qwen-sharp-v22.5.0.jinja:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
|
||||
Reference in new issue
Block a user