frank-medium: the drafter beats MTP and MTP beats nothing -- measured, not assumed

Four configurations of the same binary, identical prompts at temperature 0, the unit stopped between runs, on the deployed payload: none 17.7 tok/s card1 26.07 GB draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all four produced IDENTICAL greedy text, so the 'lossless verification' claim in the runbook is now a check rather than an assumption. MTP is not the cheaper option either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper drafting loses (n=7 accepts a third of its proposals, n=3 more than half). The two CANNOT be chained, and the failure is structural rather than tunable: 'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested but model doesn't contain MTP layers', because a chain has ONE draft context and MTP builds against whatever model that context holds -- here the DFlash2 file. The serve script therefore passes --spec-draft-model in the dflash branch ONLY, and the seam now asserts exactly that (the wiring whose violation is this error), plus that the deployment stays on the measured winner. scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking policy wants the runtime's own harness for this, and the numbers above were otherwise unreproducible one-offs.
2026-09-16 16:15:59 -07:00
parent 9a83dae73a
commit 34253a9903
4 changed files with 269 additions and 0 deletions
@@ -211,6 +211,37 @@ vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
 llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
 no proxy.

+### What the drafter is worth — and why MTP is not the alternative
+
+Measured 2026-09-16 with `scripts/bench-spec-ab.py`, four configurations of the
+same binary, identical prompts at `temperature 0`, unit stopped between runs
+(one host, one copy of a 29 GiB model). 1k-token prompt, 512-token budget:
+
+| Configuration | Decode | Drafting | VRAM (card 1) |
+|---|---:|---|---:|
+| `none` — autoregressive | 17.7 tok/s | — | 26.07 GB |
+| `draft-mtp` n_max 3 (the model's own head) | 32.4 tok/s | 58 % of 72 | 29.35 GB |
+| `draft-mtp` n_max 7 | 26.7 tok/s | 33 % of 147 | 29.65 GB |
+| **`draft-dflash` n_max 7 — deployed** | **43.5 tok/s** | 41 % of 119 | 29.42 GB |
+
+- **DFlash2 is worth 2.46× over autoregressive and 34 % over the best MTP.**
+  Deeper MTP drafting costs more than it returns: at depth 7 only a third of
+  the proposals survive, at depth 3 more than half do.
+- **All four produced identical greedy text** (`2, 3, 5, 7, 11`): speculation
+  buys speed, never a different answer. The harness warns loudly if a future
+  configuration ever breaks that.
+- **MTP is not the cheaper option.** Selecting it loads `blk.64.nextn.*` from
+  the target and adds ~3.3 GB to the card that holds it, against 2.06 GB for
+  the drafter's file — which is also why the load log says
+  `unused tensor blk.64.nextn... ignoring` under `draft-dflash`: correct
+  behaviour, not a truncated download.
+- **They cannot be chained.** `--spec-type draft-mtp,draft-dflash` dies at load
+  with `context type MTP requested but model doesn't contain MTP layers`. A
+  chain has ONE draft context, and MTP builds against whatever model that
+  context holds — here the DFlash2 file. MTP's head is loaded only when MTP *is*
+  the draft model (`creating MTP draft context against the target model ...`).
+  That is structural, not a tuning question.
+
 ### The agent-loop shape (fleet harness, 2 families × 2 turns)

 `scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two