frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0
- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards, quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on) - Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel, built on the published 0.9.3 base; docker->podman via save/load) - GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16) - Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile + sitecustomize registration); GPU_UTIL default for set -u robustness - Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir - deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults - Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback - Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6) - Runbook: MXFP6 active shape, MXFP4 rollback MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
+14
-12
@@ -1178,13 +1178,15 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance stack as a
|
||||
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
|
||||
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
|
||||
# It replaced mainline llama.cpp's layer split, which held the endpoint since
|
||||
# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a
|
||||
# third of the time-to-first-token with 4x the concurrent throughput
|
||||
# (scripts/bench-conversation.py, 2026-09-20).
|
||||
# The entrypoint is quantization-aware (config.json quant_method): the plain
|
||||
# FP8 path, the MXFP4 patch-and-serve path, and the ParoQuant MXFP6 stack
|
||||
# (PR #51's launch6_p kernel). It replaced mainline llama.cpp's layer split,
|
||||
# which held the endpoint since 2026-09-16; the conversation battery measured
|
||||
# radiance at 2.1x decode and a third of the time-to-first-token with 4x the
|
||||
# concurrent throughput (scripts/bench-conversation.py, 2026-09-20).
|
||||
#
|
||||
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
|
||||
# unit because TP=2 could not dispatch a collective on the second card (B650
|
||||
@@ -1197,13 +1199,13 @@ deploy_llm_small() {
|
||||
# pinned radiance patch clone, the patched libr4d build and the Triton compile
|
||||
# cache live outside the checkout (see the unit file's mount comments). The
|
||||
# gates below refuse a deploy that would start a container guaranteed to die.
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-ggz14/vllm-radiance-mxfp4:0.13.0-e060895}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.13.0-e060895}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-PARO-MXFP6}"
|
||||
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
|
||||
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
|
||||
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq}"
|
||||
|
||||
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
|
||||
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
|
||||
@@ -1256,8 +1258,8 @@ deploy_llm_medium() {
|
||||
# of truth), so a model swap is just "edit the quadlet + deploy". The
|
||||
# container /models mount maps to /home/dank/models on the host. The
|
||||
# drafter gate applies only to the dflash path; the patched libr4d gate
|
||||
# applies only to the mxfp4/quark path (the plain FP8 path uses the
|
||||
# image's built-in r4d.so and skips the MXFP4 patches).
|
||||
# applies to the mxfp4/quark/paroquant_mxfp6 path (the plain FP8 path uses
|
||||
# the image's built-in r4d.so and skips the MXFP4 patches).
|
||||
local quadlet_file="dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
local snap_c snap spec_method quant
|
||||
snap_c="$(sed -nE 's/^Environment=SNAP=//p' "$quadlet_file" | head -1)"
|
||||
@@ -1294,7 +1296,7 @@ deploy_llm_medium() {
|
||||
# NaNs the MXFP4 model, so the patched build is not optional there. The
|
||||
# plain FP8 path uses the image's built-in r4d.so (no MXFP4 patches).
|
||||
case "$quant" in
|
||||
mxfp4|quark)
|
||||
mxfp4|quark|paroquant_mxfp6)
|
||||
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
|
||||
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
|
||||
missing=1
|
||||
|
||||
Reference in New Issue
Block a user