diff --git a/dankserver/frank-medium/quadlet/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container index 3e503b8..0862459 100644 --- a/dankserver/frank-medium/quadlet/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -116,7 +116,7 @@ Environment=MAXLEN=262144 # through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves # image input on either checkpoint. harness models.yml keeps frank-medium # input: [text, image]. -Environment=GPU_UTIL=0.99 +Environment=GPU_UTIL=0.98 # Speculative decoding: DFlash2 external drafter at depth 7. A/B tested # against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2 # wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4) diff --git a/dankserver/frank-medium/radiance-entrypoint.sh b/dankserver/frank-medium/radiance-entrypoint.sh index 6b748c9..d0c44c1 100755 --- a/dankserver/frank-medium/radiance-entrypoint.sh +++ b/dankserver/frank-medium/radiance-entrypoint.sh @@ -44,7 +44,7 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi : "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter) : "${R4D_ATTN:=1}" : "${ASYNC:=0}" -: "${GPU_UTIL:=0.99}" +: "${GPU_UTIL:=0.98}" # Quantization-aware: read the model's weight quantization from config.json. # fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 8a03adb..c7f6172 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -33,12 +33,12 @@ single-GPU replicas, which is the shape most of this page describes. This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel, `quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on, e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and -`--gpu-memory-utilization 0.99`. The Swift variant is a finetune of +`--gpu-memory-utilization 0.98`. The Swift variant is a finetune of `ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to -`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, `GPU_UTIL` to 0.98, and the image +`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, and the image back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/ vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to 486,464 tokens with bf16 KV and held the full 262,144 window). @@ -63,9 +63,9 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | -| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~730k tokens** (estimated at 0.99 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~19.2 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` | +| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~420k tokens** (estimated at 0.98 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~17.6 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` | | Prompt cache | `--enable-prefix-caching` | -| GPU fill | `--gpu-memory-utilization 0.99` | +| GPU fill | `--gpu-memory-utilization 0.98` (the 32 GB ceiling — 0.99 fails the startup check) | | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` | | Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) | | Multimodal | `--limit-mm-per-prompt '{"image": 8}'` | @@ -73,15 +73,18 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin (`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave. -MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2), which at 0.99 fill -(34.2 GiB per card) leaves ~19.2 GiB per card for KV + activations. The -estimated pool is **~730k tokens** (the actual size is determined at boot; the -full 262,144 window fits with ~2.7x headroom). The DFlash2-FP8 drafter costs -2.1 GiB of VRAM on top. For reference the MXFP4 checkpoint (now the rollback) -auto-sized to 486,464 tokens with bf16 KV (1.86x the window), and the FP8 -checkpoint would only reach ~192,000 unquantized — which is why MXFP6's larger -weights still afford clean KV at the native 262,144 (the 0.99 fill buys the -pool back). After the first healthy start, recalibrate with `calibrate-kv.sh`. +MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2); at 0.98 fill +(31.2 GiB per card) that leaves ~17.6 GiB per card for KV + activations — an +estimated **~420k-token pool** (the actual size is set at boot; the full +262,144 window fits with ~1.6x headroom). The DFlash2-FP8 drafter costs 2.1 GiB +of VRAM on top. **0.98 is the ceiling, not a preference**: the R9700 reports +31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context + +NCCL + the radiance custom all-reduce buffers consume the 0.63 GiB gap), so a +`gpu_memory_utilization` above 0.98 (0.99 = 31.54 GiB) fails vLLM's startup +check before the weights even load. For reference the MXFP4 checkpoint (the +rollback) auto-sized to 486,464 tokens at the same 0.98 with bf16 KV (1.86x the +window); MXFP6's larger weights give a smaller pool, still comfortably above the +window. After the first healthy start, recalibrate with `calibrate-kv.sh`. ## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)