frank-medium: the drafter beats MTP and MTP beats nothing -- measured, not assumed

Four configurations of the same binary, identical prompts at temperature 0, the
unit stopped between runs, on the deployed payload:

  none            17.7 tok/s                     card1 26.07 GB
  draft-mtp  n=3  32.4 tok/s  58 % accepted       card1 29.35 GB
  draft-mtp  n=7  26.7 tok/s  33 % accepted       card1 29.65 GB
  draft-dflash 7  43.5 tok/s  41 % accepted       card1 29.42 GB

DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all
four produced IDENTICAL greedy text, so the 'lossless verification' claim in the
runbook is now a check rather than an assumption. MTP is not the cheaper option
either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper
drafting loses (n=7 accepts a third of its proposals, n=3 more than half).

The two CANNOT be chained, and the failure is structural rather than tunable:
'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested
but model doesn't contain MTP layers', because a chain has ONE draft context and
MTP builds against whatever model that context holds -- here the DFlash2 file.
The serve script therefore passes --spec-draft-model in the dflash branch ONLY,
and the seam now asserts exactly that (the wiring whose violation is this
error), plus that the deployment stays on the measured winner.

scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking
policy wants the runtime's own harness for this, and the numbers above were
otherwise unreproducible one-offs.
This commit is contained in:
Frank committed 2026-09-16 16:15:59 -07:00
1 parent 9a83dae73a
commit 34253a9903
4 files changed
+269

No files matched your search

+20
View File
@@ -73,6 +73,26 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
[ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 7 ]
}
@test "a draft MODEL is passed only for DFlash types, never for MTP" {
# MTP builds its context against the TARGET model (its head lives in that
# GGUF). Passing --spec-draft-model alongside it points the draft context at
# the drafter's file, which has no MTP layers, and the server dies at load:
# context type MTP requested but model doesn't contain MTP layers
# failed to create MTP context
# So the flag must appear exactly once, inside the dflash branch.
python3 -c "
lines = open('$SERVE').read().splitlines()
guard = next(i for i, l in enumerate(lines) if '*dflash*' in l)
indent = len(lines[guard]) - len(lines[guard].lstrip())
end = next(i for i in range(guard + 1, len(lines))
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
block = chr(10).join(lines[guard:end])
assert '--spec-draft-model' in block, 'the dflash branch must pass the draft model'
assert sum(l.count('--spec-draft-model') for l in lines) == 1, 'only that branch may pass it'
print('draft-model wiring ok')
"
}
@test "the runtime is pinned to a commit and to gfx1201 alone" {
# A floating ref would silently move the service between engine versions,
# and an unpinned arch would build for a sibling GPU that loads and then