feat(frank-medium): cut to MXFP4 checkpoint with unquantized bf16 KV

Swap the serving checkpoint back to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (quark/W4A8) and serve it with
KV_DTYPE=auto (unquantized bf16 KV) instead of the hardcoded fp8. fp8 KV
stacked on MXFP4's already-quantized W4A8 weights compounds attention-path
precision loss; clean KV tests whether that was hurting output quality. The
freed MXFP4 weights (19.37 vs 30.9 GB) buy the KV pool back, so the full
262,144 native window fits unquantized - MAXLEN returns to 262144.

The entrypoint gains a KV_DTYPE tunable (default fp8, the historical behavior),
shared across both quantization paths; the quadlet sets auto. Rollback: SNAP
back to the FP8 checkpoint with KV_DTYPE=fp8 and MAXLEN=192000.
This commit is contained in:
Frank committed 2026-09-21 16:32:19 -07:00
1 parent 0f203dfa41
commit a45f396c39
3 files changed
+29 -17

No files matched your search

@@ -36,6 +36,7 @@ set -euo pipefail
# after the weights load (2.16 GiB requested, 116 MiB free). Do not "fix"
# this into an empty expansion.
: "${KV_MEM:=18563072000}" # shipped kv-profiles.tsv row 2x7551-32624 @ 8/8192/262144/dflash
: "${KV_DTYPE:=fp8}" # fp8 = quantized KV (2x the tokens); auto = bf16 KV (half the pool)
: "${SPEC_METHOD:=dflash}"
if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
: "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
@@ -239,7 +240,7 @@ QUANT_FLAG=""
if [ "$IS_MXFP4" = "0" ]; then QUANT_FLAG="--quantization fp8"; fi
exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \
--host 0.0.0.0 --port "$PORT" \
--kv-cache-dtype fp8 --tensor-parallel-size "$TP" \
--kv-cache-dtype "$KV_DTYPE" --tensor-parallel-size "$TP" \
$QUANT_FLAG \
--gpu-memory-utilization "$GPU_UTIL" \
${KV_MEM:+--kv-cache-memory "$KV_MEM"} \