feat(frank-medium): quantization-aware entrypoint + FP8 model swap

The entrypoint now reads the model's weight quantization from config.json
and serves the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
self-speculation, ROCM_AITER unified attention with the R4D all-reduce)
or the MXFP4 patch-and-serve path. A model swap is now: change SNAP
(+ SPEC_METHOD/SPEC), restart.

- radiance-entrypoint.sh: quantization detection, gated MXFP4 env vars
  and patch block, --quantization fp8 flag
- quadlet: SNAP -> Qwen3.8-27B-Uncensored-FP8, SPEC_METHOD=mtp, SPEC=8,
  KV_MEM=0 (profile the shape, the documented plain-FP8 default)
- deploy-remote.sh: model-aware gates (read SNAP/SPEC_METHOD from the
  quadlet; drafter gate only for dflash, r4d.so gate only for mxfp4)
- tests: quantization-aware drafter/spec and KV-pool tests
- hf-fetch.sh: generic, resumable, verified, idempotent HF fetch tool
- frank-medium-fp8.manifest: 17 files (10 config + 7 shards) with sizes
  and LFS sha256
This commit is contained in:
Frank committed 2026-09-20 21:30:30 -07:00
1 parent c42780f9f3
commit d389d09d97
6 files changed
+251 -63

No files matched your search

@@ -4,13 +4,18 @@
# Unit name: frank-medium.service (container name: frank-medium)
#
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
# the ggz14/radiance-vllm-mxfp4 patches) serving
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
# It replaced the mainline llama.cpp layer-split deployment, which served the
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
# the vllm-radiance stack (stilldeadcode/vllm-radiance:0.9.3 image) serving
# orcarouter/Qwen3.8-27B-Uncensored-FP8 (FP8 E4M3 block-wise, 128x128,
# dynamic activation quantization) on the 2x Radeon AI PRO R9700 (gfx1201)
# as ONE tensor-parallel process (TP=2, R4D one-shot all-reduce) with MTP
# self-speculation (SPEC=8, the model's own single-layer MTP head).
# The entrypoint is QUANTIZATION-AWARE: it reads the model's weight
# quantization from config.json and serves the plain FP8 path (no MXFP4
# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4
# patch-and-serve path (the DFlash2 drafter). A model swap is now: change
# SNAP (+ SPEC_METHOD/SPEC), restart. The MXFP4 checkpoint
# (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq) remains on disk as the
# rollback target.
#
# SHAPE. The launcher the upstream repo ships binds the server with
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
@@ -60,7 +65,7 @@
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
[Unit]
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
Description=frank-medium — vllm-radiance FP8 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
After=network-online.target
[Container]
@@ -77,9 +82,10 @@ PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add S
# memory is not a compute target.
Environment=ROCR_VISIBLE_DEVICES=0,1
Environment=HIP_VISIBLE_DEVICES=0,1
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
# The FP8 checkpoint carries its own single-layer MTP head (mtp.layers.0),
# so MTP self-speculation needs no external drafter and no fp8_mtp.py
# rewrite step. The MXFP4 checkpoint (rollback) ships its fp8 MTP head too.
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-FP8
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
# Fleet identity: the id at /v1/models and in every response body. Every
@@ -104,19 +110,17 @@ Environment=MAXLEN=262144
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
# modelRoles vision -> frank-small).
Environment=GPU_UTIL=0.98
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
# every published number (224 t/s single-stream decode, 4749-4810 t/s
# prefill) was measured with.
Environment=SPEC_METHOD=dflash
Environment=SPEC=7
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
# explicitly because this unit does not run the launcher's lookup; valid
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
Environment=KV_MEM=18563072000
# Speculative decoding: MTP self-speculation at depth 8 (the model's own
# single-layer MTP head, applied autoregressively). The documented plain-FP8
# path (upstream docker-compose) uses num_speculative_tokens=8.
Environment=SPEC_METHOD=mtp
Environment=SPEC=8
# KV cache: 0 = profile the shape (the documented plain-FP8 path's default;
# the upstream docker-compose leaves the pin commented out). The measured pin
# for this shape is 15989735424 (14.89 GiB, upstream, 2x R9700 32GB); after
# the first healthy start, recalibrate with calibrate-kv.sh to use the extra
# VRAM on the 34.2 GiB cards.
Environment=KV_MEM=0
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
# keep parity with the launcher's rw mount).
Volume=/home/dank/models:/models