test(frank-medium): A/B DFlash2 vs MTP on FP8 — DFlash2 wins on decode

DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
This commit is contained in:
Frank committed 2026-09-20 22:20:53 -07:00
1 parent 1ae73db084
commit f998a2d9f2
6 files changed
+111 -44

No files matched your search

+27
View File
@@ -11,6 +11,33 @@ timestamp: 2026-09-14
# Bundle Change Log
## 2026-09-20 — DFlash2 vs MTP A/B on the FP8 checkpoint: DFlash2 wins on decode
- **The test.** The FP8 checkpoint carries its own single-layer MTP head, so
it can self-speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external
drafter (`SPEC_METHOD=dflash`). Both were deployed and measured on the same
FP8 target (`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`,
greedy, salted, `--burst 4`):
- **DFlash2** (depth 7): mean accept 4.24/7, geomean decode **100.0 t/s**,
TTFT 0.19 s, prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
- **MTP** (depth 8): mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s,
prefill 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
- **Verdict: DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on
TTFT and prefill.** The counterintuitive part: DFlash2 accepts *fewer*
tokens per step (4.24 vs 5.66) yet decodes *faster*. Accept length is a
proxy, not the metric — throughput is `(accept + 1) / (draft time + verify
time)`. The DFlash2 drafter is a small separate model that steps fast; the
MTP head is the target's own layer applied autoregressively 8×, so its draft
is slower. The faster step more than makes up the lower accept length.
- **FP8 + DFlash2 (100.0 t/s) beats the original MXFP4 + DFlash2 (92.4 t/s)**
— the higher-precision FP8 target verifies more of the drafter's proposals.
The drafter costs 2.1 GiB of VRAM (KV pool 1.47× vs MTP's 1.76×), an
acceptable trade for the decode win.
- **DFlash2 is the deployed shape.** The quadlet comment, the radiance
runbook (deployed-flags table, KV pool, a new A/B section), AGENTS.md
(topology row + frank-medium section), and the model-swap runbook + swap
script (both paths now default to `dflash/7`) all updated.
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,