diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index c7f6172..c994000 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -59,11 +59,11 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Setting | Value | |---|---| | Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) | -| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) | +| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; measured on the Swift finetune: mean acceptance length 7.00 / 85.7% on easy text, 5.08 / 58.3% on varied — the drafter survives the base-model change. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) | | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | -| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~420k tokens** (estimated at 0.98 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~17.6 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` | +| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 425,047 tokens** (measured at 0.98 fill: 14.42 GiB per card available for KV, 1.62x the 262,144 window); rollback: `KV_DTYPE=fp8` | | Prompt cache | `--enable-prefix-caching` | | GPU fill | `--gpu-memory-utilization 0.98` (the 32 GB ceiling — 0.99 fails the startup check) | | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` | @@ -74,10 +74,10 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin (`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave. MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2); at 0.98 fill -(31.2 GiB per card) that leaves ~17.6 GiB per card for KV + activations — an -estimated **~420k-token pool** (the actual size is set at boot; the full -262,144 window fits with ~1.6x headroom). The DFlash2-FP8 drafter costs 2.1 GiB -of VRAM on top. **0.98 is the ceiling, not a preference**: the R9700 reports +(31.22 GiB per card) that leaves **14.42 GiB per card for KV** — a measured +**425,047-token pool**, 1.62x the 262,144 window (boot 2026-09-22). The +DFlash2-FP8 drafter costs 2.1 GiB of VRAM on top. +**0.98 is the ceiling, not a preference**: the R9700 reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context + NCCL + the radiance custom all-reduce buffers consume the 0.63 GiB gap), so a `gpu_memory_utilization` above 0.98 (0.99 = 31.54 GiB) fails vLLM's startup