frank-medium: revert GPU_UTIL to 0.98 (0.99 is infeasible on 32 GB R9700)
0.99 failed the vLLM startup memory check before weight load: the R9700 reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so 0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22 GiB) is the ceiling and the prior production value. Also fixed the runbook KV math that had shipped with the 0.99 change: the ~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's 19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the 34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98 fill.
This commit is contained in:
1 parent
e9a01cd013
commit
d5c8e408b3
3 files changed
+18
-15
No files matched your search
@@ -44,7 +44,7 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
|
||||
: "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
|
||||
: "${R4D_ATTN:=1}"
|
||||
: "${ASYNC:=0}"
|
||||
: "${GPU_UTIL:=0.99}"
|
||||
: "${GPU_UTIL:=0.98}"
|
||||
|
||||
# Quantization-aware: read the model's weight quantization from config.json.
|
||||
# fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
|
||||
|
||||
Reference in new issue
Block a user