frank-medium: revert GPU_UTIL to 0.98 (0.99 is infeasible on 32 GB R9700)

0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.

Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
This commit is contained in:
Frank committed 2026-09-22 00:15:06 -07:00
1 parent e9a01cd013
commit d5c8e408b3
3 files changed
+18 -15

No files matched your search

@@ -116,7 +116,7 @@ Environment=MAXLEN=262144
# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
# image input on either checkpoint. harness models.yml keeps frank-medium
# input: [text, image].
Environment=GPU_UTIL=0.99
Environment=GPU_UTIL=0.98
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2
# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4)