frank-medium: revert GPU_UTIL to 0.98 (0.99 is infeasible on 32 GB R9700)
0.99 failed the vLLM startup memory check before weight load: the R9700 reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so 0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22 GiB) is the ceiling and the prior production value. Also fixed the runbook KV math that had shipped with the 0.99 change: the ~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's 19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the 34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98 fill.
This commit is contained in:
1 parent
e9a01cd013
commit
d5c8e408b3
3 files changed
+18
-15
No files matched your search
@@ -116,7 +116,7 @@ Environment=MAXLEN=262144
|
||||
# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
|
||||
# image input on either checkpoint. harness models.yml keeps frank-medium
|
||||
# input: [text, image].
|
||||
Environment=GPU_UTIL=0.99
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
|
||||
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2
|
||||
# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4)
|
||||
|
||||
Reference in new issue
Block a user