frank-medium: revert GPU_UTIL to 0.98 (0.99 is infeasible on 32 GB R9700)

0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.

Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
This commit is contained in:
Frank committed 2026-09-22 00:15:06 -07:00
1 parent e9a01cd013
commit d5c8e408b3
3 files changed
+18 -15

No files matched your search

@@ -44,7 +44,7 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
: "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
: "${R4D_ATTN:=1}"
: "${ASYNC:=0}"
: "${GPU_UTIL:=0.99}"
: "${GPU_UTIL:=0.98}"
# Quantization-aware: read the model's weight quantization from config.json.
# fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP