feat(frank-medium): quantization-aware entrypoint + FP8 model swap
The entrypoint now reads the model's weight quantization from config.json and serves the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP self-speculation, ROCM_AITER unified attention with the R4D all-reduce) or the MXFP4 patch-and-serve path. A model swap is now: change SNAP (+ SPEC_METHOD/SPEC), restart. - radiance-entrypoint.sh: quantization detection, gated MXFP4 env vars and patch block, --quantization fp8 flag - quadlet: SNAP -> Qwen3.8-27B-Uncensored-FP8, SPEC_METHOD=mtp, SPEC=8, KV_MEM=0 (profile the shape, the documented plain-FP8 default) - deploy-remote.sh: model-aware gates (read SNAP/SPEC_METHOD from the quadlet; drafter gate only for dflash, r4d.so gate only for mxfp4) - tests: quantization-aware drafter/spec and KV-pool tests - hf-fetch.sh: generic, resumable, verified, idempotent HF fetch tool - frank-medium-fp8.manifest: 17 files (10 config + 7 shards) with sizes and LFS sha256
This commit is contained in:
1 parent
c42780f9f3
commit
d389d09d97
6 files changed
+251
-63
No files matched your search
@@ -4,13 +4,18 @@
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
|
||||
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
|
||||
# the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
||||
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
|
||||
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
|
||||
# It replaced the mainline llama.cpp layer-split deployment, which served the
|
||||
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
|
||||
# the vllm-radiance stack (stilldeadcode/vllm-radiance:0.9.3 image) serving
|
||||
# orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 128x128,
|
||||
# dynamic activation quantization) on the 2x Radeon AI PRO R9700 (gfx1201)
|
||||
# as ONE tensor-parallel process (TP=2, R4D one-shot all-reduce) with MTP
|
||||
# self-speculation (SPEC=8, the model's own single-layer MTP head).
|
||||
# The entrypoint is QUANTIZATION-AWARE: it reads the model's weight
|
||||
# quantization from config.json and serves the plain FP8 path (no MXFP4
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4
|
||||
# patch-and-serve path (the DFlash2 drafter). A model swap is now: change
|
||||
# SNAP (+ SPEC_METHOD/SPEC), restart. The MXFP4 checkpoint
|
||||
# (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq) remains on disk as the
|
||||
# rollback target.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
@@ -60,7 +65,7 @@
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
Description=frank-medium — vllm-radiance FP8 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
@@ -77,9 +82,10 @@ PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add S
|
||||
# memory is not a compute target.
|
||||
Environment=ROCR_VISIBLE_DEVICES=0,1
|
||||
Environment=HIP_VISIBLE_DEVICES=0,1
|
||||
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
|
||||
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
# The FP8 checkpoint carries its own single-layer MTP head (mtp.layers.0),
|
||||
# so MTP self-speculation needs no external drafter and no fp8_mtp.py
|
||||
# rewrite step. The MXFP4 checkpoint (rollback) ships its fp8 MTP head too.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-FP8
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
# Fleet identity: the id at /v1/models and in every response body. Every
|
||||
@@ -104,19 +110,17 @@ Environment=MAXLEN=262144
|
||||
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
||||
# modelRoles vision -> frank-small).
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
||||
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
||||
# prefill) was measured with.
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
|
||||
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
|
||||
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
|
||||
# explicitly because this unit does not run the launcher's lookup; valid
|
||||
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
|
||||
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
|
||||
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
|
||||
Environment=KV_MEM=18563072000
|
||||
# Speculative decoding: MTP self-speculation at depth 8 (the model's own
|
||||
# single-layer MTP head, applied autoregressively). The documented plain-FP8
|
||||
# path (upstream docker-compose) uses num_speculative_tokens=8.
|
||||
Environment=SPEC_METHOD=mtp
|
||||
Environment=SPEC=8
|
||||
# KV cache: 0 = profile the shape (the documented plain-FP8 path's default;
|
||||
# the upstream docker-compose leaves the pin commented out). The measured pin
|
||||
# for this shape is 15989735424 (14.89 GiB, upstream, 2x R9700 32GB); after
|
||||
# the first healthy start, recalibrate with calibrate-kv.sh to use the extra
|
||||
# VRAM on the 34.2 GiB cards.
|
||||
Environment=KV_MEM=0
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
|
||||
# keep parity with the launcher's rw mount).
|
||||
Volume=/home/dank/models:/models
|
||||
|
||||
Reference in new issue
Block a user