frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0
- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards, quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on) - Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel, built on the published 0.9.3 base; docker->podman via save/load) - GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16) - Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile + sitecustomize registration); GPU_UTIL default for set -u robustness - Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir - deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults - Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback - Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6) - Runbook: MXFP6 active shape, MXFP4 rollback MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
1 parent
600812d459
commit
e9a01cd013
6 files changed
+709
-111
No files matched your search
@@ -95,10 +95,11 @@ env_str() {
|
||||
# DFlash2 drafts at most 7 tokens per step; deeper is a different
|
||||
# drafter's contract and costs acceptance, not just VRAM.
|
||||
[ "$spec" -eq 7 ]
|
||||
# The MXFP4 decode kernel band covers MAXSEQS x (SPEC+1) rows, and the
|
||||
# shipped default covers 64. Exceeding it without the entrypoint's >8
|
||||
# bump is a launch failure at the worst moment (first concurrent
|
||||
# request).
|
||||
# The MXFP4 / ParoQuant-MXFP6 decode kernel band covers MAXSEQS x
|
||||
# (SPEC+1) rows, and the shipped default covers 64 (the mxfp6 profile
|
||||
# block exports the same RADIANCE_MXFP4_DECODE_MAX_M=64). Exceeding it
|
||||
# without the entrypoint's >8 bump is a launch failure at the worst
|
||||
# moment (first concurrent request).
|
||||
[ $(( maxseqs * (spec + 1) )) -le 64 ]
|
||||
grep -q 'RADIANCE_MXFP4_DECODE_MAX_M' "$ENTRY"
|
||||
;;
|
||||
|
||||
Reference in new issue
Block a user