feat(frank-medium): cut to MXFP4 checkpoint with unquantized bf16 KV

Swap the serving checkpoint back to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (quark/W4A8) and serve it with
KV_DTYPE=auto (unquantized bf16 KV) instead of the hardcoded fp8. fp8 KV
stacked on MXFP4's already-quantized W4A8 weights compounds attention-path
precision loss; clean KV tests whether that was hurting output quality. The
freed MXFP4 weights (19.37 vs 30.9 GB) buy the KV pool back, so the full
262,144 native window fits unquantized - MAXLEN returns to 262144.

The entrypoint gains a KV_DTYPE tunable (default fp8, the historical behavior),
shared across both quantization paths; the quadlet sets auto. Rollback: SNAP
back to the FP8 checkpoint with KV_DTYPE=fp8 and MAXLEN=192000.
This commit is contained in:
Frank committed 2026-09-21 16:32:19 -07:00
1 parent 0f203dfa41
commit a45f396c39
3 files changed
+29 -17

No files matched your search

+6 -1
View File
@@ -114,19 +114,24 @@ env_str() {
}
@test "the KV pool holds a full window and fits beside the weights" {
local kv_mem maxseqs chunk maxlen spec
local kv_mem maxseqs chunk maxlen spec kv_dtype
kv_mem="$(env_value KV_MEM)"
maxseqs="$(env_value MAXSEQS)"
chunk="$(env_value CHUNK)"
maxlen="$(env_value MAXLEN)"
spec="$(env_value SPEC)"
kv_dtype="$(env_str KV_DTYPE)"
[ -n "$kv_mem" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ] && [ -n "$maxlen" ]
# KV_MEM=0 = profile the shape (the documented plain-FP8 path's default;
# the upstream docker-compose leaves the pin commented out). The pool is
# auto-sized to fit, so there is no pin to verify — only that the shape
# is set and the model's weights fit beside an auto-sized pool.
if [ "$kv_mem" -eq 0 ]; then
# MXFP4 + unquantized (KV_DTYPE=auto -> bf16) KV: MXFP4's freed weights
# (19.37 vs 30.9 GB) buy the pool back, so the full 262,144 native window
# still fits clean (~290k f16 pool); vLLM boots or the pool is too small.
[ "$maxseqs" -eq 8 ] && [ "$chunk" -eq 8192 ] && [ "$maxlen" -eq 262144 ]
[ "$kv_dtype" = "auto" ]
return 0
fi
# Pinned pool: the pin must hold at least ONE full-window request or the