dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: record measured MXFP6 boot shape (KV pool, drafter acceptance)

Browse Source 
Boot 2026-09-22 at 0.98 fill: KV pool auto-sized to 425,047 tokens (1.62x
the 262,144 window) with 14.42 GiB per card available for KV. The DFlash2
drafter on the Swift finetune measures mean acceptance length 7.00 / 85.7%
(easy) and 5.08 / 58.3% (varied) — it survives the base-model change.

Replaces the estimated ~420k / ~17.6 GiB figures with the measured values.
This commit is contained in:
Frank
2026-09-22 00:44:52 -07:00
parent d5c8e408b3
commit 24f7a7772d
1 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+6 -6
View File
@@ -59,11 +59,11 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Setting | Value |
|---|---|
| Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; measured on the Swift finetune: mean acceptance length 7.00 / 85.7% on easy text, 5.08 / 58.3% on varied — the drafter survives the base-model change. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~420k tokens** (estimated at 0.98 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~17.6 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 425,047 tokens** (measured at 0.98 fill: 14.42 GiB per card available for KV, 1.62x the 262,144 window); rollback: `KV_DTYPE=fp8` |
| Prompt cache | `--enable-prefix-caching` |
| GPU fill | `--gpu-memory-utilization 0.98` (the 32 GB ceiling — 0.99 fails the startup check) |
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
@@ -74,10 +74,10 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2); at 0.98 fill
(31.2 GiB per card) that leaves ~17.6 GiB per card for KV + activations — an
estimated **~420k-token pool** (the actual size is set at boot; the full
262,144 window fits with ~1.6x headroom). The DFlash2-FP8 drafter costs 2.1 GiB
of VRAM on top. **0.98 is the ceiling, not a preference**: the R9700 reports
(31.22 GiB per card) that leaves **14.42 GiB per card for KV** — a measured
**425,047-token pool**, 1.62x the 262,144 window (boot 2026-09-22). The
DFlash2-FP8 drafter costs 2.1 GiB of VRAM on top.
**0.98 is the ceiling, not a preference**: the R9700 reports
31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context +
NCCL + the radiance custom all-reduce buffers consume the 0.63 GiB gap), so a
`gpu_memory_utilization` above 0.98 (0.99 = 31.54 GiB) fails vLLM's startup