feat(frank-medium): cut to MXFP4 checkpoint with unquantized bf16 KV
Swap the serving checkpoint back to just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (quark/W4A8) and serve it with KV_DTYPE=auto (unquantized bf16 KV) instead of the hardcoded fp8. fp8 KV stacked on MXFP4's already-quantized W4A8 weights compounds attention-path precision loss; clean KV tests whether that was hurting output quality. The freed MXFP4 weights (19.37 vs 30.9 GB) buy the KV pool back, so the full 262,144 native window fits unquantized - MAXLEN returns to 262144. The entrypoint gains a KV_DTYPE tunable (default fp8, the historical behavior), shared across both quantization paths; the quadlet sets auto. Rollback: SNAP back to the FP8 checkpoint with KV_DTYPE=fp8 and MAXLEN=192000.
This commit is contained in:
1 parent
0f203dfa41
commit
a45f396c39
3 files changed
+29
-17
No files matched your search
@@ -36,6 +36,7 @@ set -euo pipefail
|
||||
# after the weights load (2.16 GiB requested, 116 MiB free). Do not "fix"
|
||||
# this into an empty expansion.
|
||||
: "${KV_MEM:=18563072000}" # shipped kv-profiles.tsv row 2x7551-32624 @ 8/8192/262144/dflash
|
||||
: "${KV_DTYPE:=fp8}" # fp8 = quantized KV (2x the tokens); auto = bf16 KV (half the pool)
|
||||
: "${SPEC_METHOD:=dflash}"
|
||||
if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
|
||||
: "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
|
||||
@@ -239,7 +240,7 @@ QUANT_FLAG=""
|
||||
if [ "$IS_MXFP4" = "0" ]; then QUANT_FLAG="--quantization fp8"; fi
|
||||
exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \
|
||||
--host 0.0.0.0 --port "$PORT" \
|
||||
--kv-cache-dtype fp8 --tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype "$KV_DTYPE" --tensor-parallel-size "$TP" \
|
||||
$QUANT_FLAG \
|
||||
--gpu-memory-utilization "$GPU_UTIL" \
|
||||
${KV_MEM:+--kv-cache-memory "$KV_MEM"} \
|
||||
|
||||
Reference in new issue
Block a user