feat(frank-medium): quantization-aware entrypoint + FP8 model swap
The entrypoint now reads the model's weight quantization from config.json and serves the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP self-speculation, ROCM_AITER unified attention with the R4D all-reduce) or the MXFP4 patch-and-serve path. A model swap is now: change SNAP (+ SPEC_METHOD/SPEC), restart. - radiance-entrypoint.sh: quantization detection, gated MXFP4 env vars and patch block, --quantization fp8 flag - quadlet: SNAP -> Qwen3.8-27B-Uncensored-FP8, SPEC_METHOD=mtp, SPEC=8, KV_MEM=0 (profile the shape, the documented plain-FP8 default) - deploy-remote.sh: model-aware gates (read SNAP/SPEC_METHOD from the quadlet; drafter gate only for dflash, r4d.so gate only for mxfp4) - tests: quantization-aware drafter/spec and KV-pool tests - hf-fetch.sh: generic, resumable, verified, idempotent HF fetch tool - frank-medium-fp8.manifest: 17 files (10 config + 7 shards) with sizes and LFS sha256
This commit is contained in:
Frank
@@ -4,13 +4,18 @@
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
|
||||
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
|
||||
# the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
||||
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
|
||||
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
|
||||
# It replaced the mainline llama.cpp layer-split deployment, which served the
|
||||
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
|
||||
# the vllm-radiance stack (stilldeadcode/vllm-radiance:0.9.3 image) serving
|
||||
# orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 128x128,
|
||||
# dynamic activation quantization) on the 2x Radeon AI PRO R9700 (gfx1201)
|
||||
# as ONE tensor-parallel process (TP=2, R4D one-shot all-reduce) with MTP
|
||||
# self-speculation (SPEC=8, the model's own single-layer MTP head).
|
||||
# The entrypoint is QUANTIZATION-AWARE: it reads the model's weight
|
||||
# quantization from config.json and serves the plain FP8 path (no MXFP4
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4
|
||||
# patch-and-serve path (the DFlash2 drafter). A model swap is now: change
|
||||
# SNAP (+ SPEC_METHOD/SPEC), restart. The MXFP4 checkpoint
|
||||
# (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq) remains on disk as the
|
||||
# rollback target.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
@@ -60,7 +65,7 @@
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
Description=frank-medium — vllm-radiance FP8 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
@@ -77,9 +82,10 @@ PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add S
|
||||
# memory is not a compute target.
|
||||
Environment=ROCR_VISIBLE_DEVICES=0,1
|
||||
Environment=HIP_VISIBLE_DEVICES=0,1
|
||||
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
|
||||
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
# The FP8 checkpoint carries its own single-layer MTP head (mtp.layers.0),
|
||||
# so MTP self-speculation needs no external drafter and no fp8_mtp.py
|
||||
# rewrite step. The MXFP4 checkpoint (rollback) ships its fp8 MTP head too.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-FP8
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
# Fleet identity: the id at /v1/models and in every response body. Every
|
||||
@@ -104,19 +110,17 @@ Environment=MAXLEN=262144
|
||||
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
||||
# modelRoles vision -> frank-small).
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
||||
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
||||
# prefill) was measured with.
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
|
||||
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
|
||||
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
|
||||
# explicitly because this unit does not run the launcher's lookup; valid
|
||||
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
|
||||
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
|
||||
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
|
||||
Environment=KV_MEM=18563072000
|
||||
# Speculative decoding: MTP self-speculation at depth 8 (the model's own
|
||||
# single-layer MTP head, applied autoregressively). The documented plain-FP8
|
||||
# path (upstream docker-compose) uses num_speculative_tokens=8.
|
||||
Environment=SPEC_METHOD=mtp
|
||||
Environment=SPEC=8
|
||||
# KV cache: 0 = profile the shape (the documented plain-FP8 path's default;
|
||||
# the upstream docker-compose leaves the pin commented out). The measured pin
|
||||
# for this shape is 15989735424 (14.89 GiB, upstream, 2x R9700 32GB); after
|
||||
# the first healthy start, recalibrate with calibrate-kv.sh to use the extra
|
||||
# VRAM on the 34.2 GiB cards.
|
||||
Environment=KV_MEM=0
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
|
||||
# keep parity with the launcher's rw mount).
|
||||
Volume=/home/dank/models:/models
|
||||
|
||||
@@ -42,7 +42,20 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
|
||||
: "${R4D_ATTN:=1}"
|
||||
: "${ASYNC:=0}"
|
||||
|
||||
if [ "$R4D_ATTN" = "1" ]; then ATTN=R4D; else ATTN=ROCM_AITER_UNIFIED_ATTN; fi
|
||||
# Quantization-aware: read the model's weight quantization from config.json.
|
||||
# fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
|
||||
# self-speculation, ROCM_AITER unified attention with the R4D all-reduce).
|
||||
# mxfp4/quark (or unknown) -> the MXFP4 patch-and-serve path (the default).
|
||||
QUANT="$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("quantization_config",{}).get("quant_method",""))' "$SNAP/config.json" 2>/dev/null || true)"
|
||||
case "$QUANT" in
|
||||
fp8) IS_MXFP4=0 ;;
|
||||
mxfp4|quark|"") IS_MXFP4=1 ;;
|
||||
*) IS_MXFP4=1; echo "[radiance] WARNING: quantization '$QUANT'; defaulting to the MXFP4 path" >&2 ;;
|
||||
esac
|
||||
|
||||
if [ "$IS_MXFP4" = "0" ]; then ATTN=ROCM_AITER_UNIFIED_ATTN
|
||||
elif [ "$R4D_ATTN" = "1" ]; then ATTN=R4D
|
||||
else ATTN=ROCM_AITER_UNIFIED_ATTN; fi
|
||||
if [ "$ASYNC" = 1 ]; then ASYNC_FLAG="--async-scheduling"; UNPAD=false; else
|
||||
ASYNC_FLAG="--no-async-scheduling"; UNPAD=true; fi
|
||||
|
||||
@@ -82,7 +95,11 @@ export RADIANCE_DFLASH_SELECTOR_TOPK="${RADIANCE_DFLASH_SELECTOR_TOPK:-}"
|
||||
export RADIANCE_R4D_REPORT=1
|
||||
export RADIANCE_PRESHUFFLE=${RADIANCE_PRESHUFFLE:-1}
|
||||
export RADIANCE_MXFP4_SANITIZE="${RADIANCE_MXFP4_SANITIZE:-0}"
|
||||
export RADIANCE_MXFP4=1 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_W4A8_MIN_M="$MIN_M"
|
||||
if [ "$IS_MXFP4" = "1" ]; then
|
||||
export RADIANCE_MXFP4=1 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_W4A8_MIN_M="$MIN_M"
|
||||
else
|
||||
export RADIANCE_MXFP4=0 RADIANCE_MXFP4_W4A8=0 RADIANCE_USE_R4D_AR_QUANT=1
|
||||
fi
|
||||
export RADIANCE_FAST_DRAFT=${RADIANCE_FAST_DRAFT:-1}
|
||||
export RADIANCE_DRAFT_TAU=${RADIANCE_DRAFT_TAU:-0.20}
|
||||
export RADIANCE_DRAFT_RERANK
|
||||
@@ -171,6 +188,7 @@ if [ -n "$CC_ITEMS" ]; then
|
||||
EXTRA="$EXTRA --compilation-config {$CC_ITEMS}"
|
||||
fi
|
||||
|
||||
if [ "$IS_MXFP4" = "1" ]; then
|
||||
# ---------------------------------------------------------------- patch block
|
||||
# Byte-for-byte the launcher's container-start block. The patches come from
|
||||
# the /patches mount (the pinned radiance clone). SP must match the image's
|
||||
@@ -214,9 +232,15 @@ fi
|
||||
# Leave /patches before exec: a stale radiance_mxfp4_fp8.so there would shadow
|
||||
# the one just compiled into site-packages (cwd precedes it on sys.path).
|
||||
cd /
|
||||
fi
|
||||
# The weight-quantization flag. MXFP4 is auto-detected from the checkpoint;
|
||||
# FP8 is passed explicitly (the documented plain-FP8 path).
|
||||
QUANT_FLAG=""
|
||||
if [ "$IS_MXFP4" = "0" ]; then QUANT_FLAG="--quantization fp8"; fi
|
||||
exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \
|
||||
--host 0.0.0.0 --port "$PORT" \
|
||||
--kv-cache-dtype fp8 --tensor-parallel-size "$TP" \
|
||||
$QUANT_FLAG \
|
||||
--gpu-memory-utilization "$GPU_UTIL" \
|
||||
${KV_MEM:+--kv-cache-memory "$KV_MEM"} \
|
||||
--max-model-len "$MAXLEN" --max-num-seqs "$MAXSEQS" --max-num-batched-tokens "$CHUNK" \
|
||||
|
||||
Reference in New Issue
Block a user