frank-medium: the drafter beats MTP and MTP beats nothing -- measured, not assumed

Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:

  none            17.7 tok/s                     card1 26.07 GB
  draft-mtp  n=3  32.4 tok/s  58 % accepted       card1 29.35 GB
  draft-mtp  n=7  26.7 tok/s  33 % accepted       card1 29.65 GB
  draft-dflash 7  43.5 tok/s  41 % accepted       card1 29.42 GB

DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).

The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.

scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
This commit is contained in:
Frank committed 2026-09-16 16:15:59 -07:00
1 parent 9a83dae73a
commit 34253a9903
4 files changed
+269

No files matched your search

+26
View File
@@ -149,6 +149,32 @@ SPEC_DRAFT_DEVICE=ROCm1
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16
# MTP vs the drafter, MEASURED 2026-09-16 (temperature 0, 1k prompt / 512 out;
# the autoregressive baseline and both MTP depths were run against the deployed
# configuration in one session, same binary, same prompts):
#
# none 17.7 tok/s card1 26.07 GB
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
#
# All four produced IDENTICAL greedy text, so verification is lossless whichever
# proposer runs. DFlash2 is worth 2.46x over autoregressive and 34 % over the
# best MTP; deeper MTP drafting costs more than it returns (n=7 accepts 33 % of
# 147 proposals, n=3 accepts 58 % of 72). The MTP head is not free either:
# selecting it loads blk.64.nextn.* and adds ~3.3 GB to the card that holds it,
# against 2.06 GB for the drafter's file. Do not switch this without
# re-measuring.
#
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
# context type MTP requested but model doesn't contain MTP layers
# failed to create MTP context
# A chain has ONE draft context, and MTP builds against whatever model that
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
# only loaded when MTP IS the draft model ("creating MTP draft context against
# the target model ..."), so a chain of the two is a structural dead end, not a
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
# Text only: the server must not autoload a vision tower. Vision stays on
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
NO_MMPROJ=1