dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: revert GPU_UTIL to 0.98 (0.99 is infeasible on 32 GB R9700)

Browse Source 
0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.

Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
This commit is contained in:
Frank
2026-09-22 00:15:06 -07:00
parent e9a01cd013
commit d5c8e408b3
3 changed files with 18 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+16 -13
View File
@@ -33,12 +33,12 @@ single-GPU replicas, which is the shape most of this page describes.
This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel,
`quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on,
e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and
`--gpu-memory-utilization 0.99`. The Swift variant is a finetune of
`--gpu-memory-utilization 0.98`. The Swift variant is a finetune of
`ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the
DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may
drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec
decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to
`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, `GPU_UTIL` to 0.98, and the image
`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, and the image
back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/
vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to
486,464 tokens with bf16 KV and held the full 262,144 window).
@@ -63,9 +63,9 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~730k tokens** (estimated at 0.99 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~19.2 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~420k tokens** (estimated at 0.98 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~17.6 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` |
| Prompt cache | `--enable-prefix-caching` |
| GPU fill | `--gpu-memory-utilization 0.99` |
| GPU fill | `--gpu-memory-utilization 0.98` (the 32 GB ceiling — 0.99 fails the startup check) |
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
| Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) |
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
@@ -73,15 +73,18 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2), which at 0.99 fill
(34.2 GiB per card) leaves ~19.2 GiB per card for KV + activations. The
estimated pool is **~730k tokens** (the actual size is determined at boot; the
full 262,144 window fits with ~2.7x headroom). The DFlash2-FP8 drafter costs
2.1 GiB of VRAM on top. For reference the MXFP4 checkpoint (now the rollback)
auto-sized to 486,464 tokens with bf16 KV (1.86x the window), and the FP8
checkpoint would only reach ~192,000 unquantized — which is why MXFP6's larger
weights still afford clean KV at the native 262,144 (the 0.99 fill buys the
pool back). After the first healthy start, recalibrate with `calibrate-kv.sh`.
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2); at 0.98 fill
(31.2 GiB per card) that leaves ~17.6 GiB per card for KV + activations — an
estimated **~420k-token pool** (the actual size is set at boot; the full
262,144 window fits with ~1.6x headroom). The DFlash2-FP8 drafter costs 2.1 GiB
of VRAM on top. **0.98 is the ceiling, not a preference**: the R9700 reports
31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context +
NCCL + the radiance custom all-reduce buffers consume the 0.63 GiB gap), so a
`gpu_memory_utilization` above 0.98 (0.99 = 31.54 GiB) fails vLLM's startup
check before the weights even load. For reference the MXFP4 checkpoint (the
rollback) auto-sized to 486,464 tokens at the same 0.98 with bf16 KV (1.86x the
window); MXFP6's larger weights give a smaller pool, still comfortably above the
window. After the first healthy start, recalibrate with `calibrate-kv.sh`.
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)