feat(frank-medium): cut to MXFP4 checkpoint with unquantized bf16 KV
Swap the serving checkpoint back to just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (quark/W4A8) and serve it with KV_DTYPE=auto (unquantized bf16 KV) instead of the hardcoded fp8. fp8 KV stacked on MXFP4's already-quantized W4A8 weights compounds attention-path precision loss; clean KV tests whether that was hurting output quality. The freed MXFP4 weights (19.37 vs 30.9 GB) buy the KV pool back, so the full 262,144 native window fits unquantized - MAXLEN returns to 262144. The entrypoint gains a KV_DTYPE tunable (default fp8, the historical behavior), shared across both quantization paths; the quadlet sets auto. Rollback: SNAP back to the FP8 checkpoint with KV_DTYPE=fp8 and MAXLEN=192000.
This commit is contained in:
1 parent
0f203dfa41
commit
a45f396c39
3 files changed
+29
-17
No files matched your search
@@ -114,19 +114,24 @@ env_str() {
|
||||
}
|
||||
|
||||
@test "the KV pool holds a full window and fits beside the weights" {
|
||||
local kv_mem maxseqs chunk maxlen spec
|
||||
local kv_mem maxseqs chunk maxlen spec kv_dtype
|
||||
kv_mem="$(env_value KV_MEM)"
|
||||
maxseqs="$(env_value MAXSEQS)"
|
||||
chunk="$(env_value CHUNK)"
|
||||
maxlen="$(env_value MAXLEN)"
|
||||
spec="$(env_value SPEC)"
|
||||
kv_dtype="$(env_str KV_DTYPE)"
|
||||
[ -n "$kv_mem" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ] && [ -n "$maxlen" ]
|
||||
# KV_MEM=0 = profile the shape (the documented plain-FP8 path's default;
|
||||
# the upstream docker-compose leaves the pin commented out). The pool is
|
||||
# auto-sized to fit, so there is no pin to verify — only that the shape
|
||||
# is set and the model's weights fit beside an auto-sized pool.
|
||||
if [ "$kv_mem" -eq 0 ]; then
|
||||
# MXFP4 + unquantized (KV_DTYPE=auto -> bf16) KV: MXFP4's freed weights
|
||||
# (19.37 vs 30.9 GB) buy the pool back, so the full 262,144 native window
|
||||
# still fits clean (~290k f16 pool); vLLM boots or the pool is too small.
|
||||
[ "$maxseqs" -eq 8 ] && [ "$chunk" -eq 8192 ] && [ "$maxlen" -eq 262144 ]
|
||||
[ "$kv_dtype" = "auto" ]
|
||||
return 0
|
||||
fi
|
||||
# Pinned pool: the pin must hold at least ONE full-window request or the
|
||||
|
||||
Reference in new issue
Block a user