frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0

- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards,
  quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on)
- Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel,
  built on the published 0.9.3 base; docker->podman via save/load)
- GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16)
- Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile +
  sitecustomize registration); GPU_UTIL default for set -u robustness
- Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir
- deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults
- Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback
- Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6)
- Runbook: MXFP6 active shape, MXFP4 rollback

MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
Frank
2026-09-21 22:46:44 -07:00
parent 600812d459
commit e9a01cd013
6 changed files with 709 additions and 111 deletions
+14 -12
View File
@@ -1178,13 +1178,15 @@ deploy_llm_small() {
wait_for_health frank-small http://127.0.0.1:7771/health 900
}
# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a
# deploy_llm_medium — frank-medium is the vllm-radiance stack as a
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
# It replaced mainline llama.cpp's layer split, which held the endpoint since
# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a
# third of the time-to-first-token with 4x the concurrent throughput
# (scripts/bench-conversation.py, 2026-09-20).
# The entrypoint is quantization-aware (config.json quant_method): the plain
# FP8 path, the MXFP4 patch-and-serve path, and the ParoQuant MXFP6 stack
# (PR #51's launch6_p kernel). It replaced mainline llama.cpp's layer split,
# which held the endpoint since 2026-09-16; the conversation battery measured
# radiance at 2.1x decode and a third of the time-to-first-token with 4x the
# concurrent throughput (scripts/bench-conversation.py, 2026-09-20).
#
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
# unit because TP=2 could not dispatch a collective on the second card (B650
@@ -1197,13 +1199,13 @@ deploy_llm_small() {
# pinned radiance patch clone, the patched libr4d build and the Triton compile
# cache live outside the checkout (see the unit file's mount comments). The
# gates below refuse a deploy that would start a container guaranteed to die.
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}"
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}"
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-ggz14/vllm-radiance-mxfp4:0.13.0-e060895}"
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.13.0-e060895}"
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-PARO-MXFP6}"
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}"
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq}"
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
@@ -1256,8 +1258,8 @@ deploy_llm_medium() {
# of truth), so a model swap is just "edit the quadlet + deploy". The
# container /models mount maps to /home/dank/models on the host. The
# drafter gate applies only to the dflash path; the patched libr4d gate
# applies only to the mxfp4/quark path (the plain FP8 path uses the
# image's built-in r4d.so and skips the MXFP4 patches).
# applies to the mxfp4/quark/paroquant_mxfp6 path (the plain FP8 path uses
# the image's built-in r4d.so and skips the MXFP4 patches).
local quadlet_file="dankserver/frank-medium/quadlet/frank-medium.container"
local snap_c snap spec_method quant
snap_c="$(sed -nE 's/^Environment=SNAP=//p' "$quadlet_file" | head -1)"
@@ -1294,7 +1296,7 @@ deploy_llm_medium() {
# NaNs the MXFP4 model, so the patched build is not optional there. The
# plain FP8 path uses the image's built-in r4d.so (no MXFP4 patches).
case "$quant" in
mxfp4|quark)
mxfp4|quark|paroquant_mxfp6)
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
missing=1