frank-medium: the drafter beats MTP and MTP beats nothing -- measured, not assumed
Four configurations of the same binary, identical prompts at temperature 0, the unit stopped between runs, on the deployed payload: none 17.7 tok/s card1 26.07 GB draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all four produced IDENTICAL greedy text, so the 'lossless verification' claim in the runbook is now a check rather than an assumption. MTP is not the cheaper option either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper drafting loses (n=7 accepts a third of its proposals, n=3 more than half). The two CANNOT be chained, and the failure is structural rather than tunable: 'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested but model doesn't contain MTP layers', because a chain has ONE draft context and MTP builds against whatever model that context holds -- here the DFlash2 file. The serve script therefore passes --spec-draft-model in the dflash branch ONLY, and the seam now asserts exactly that (the wiring whose violation is this error), plus that the deployment stays on the measured winner. scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking policy wants the runtime's own harness for this, and the numbers above were otherwise unreproducible one-offs.
This commit is contained in:
1 parent
9a83dae73a
commit
34253a9903
4 files changed
+269
No files matched your search
@@ -149,6 +149,32 @@ SPEC_DRAFT_DEVICE=ROCm1
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
SPEC_DRAFT_TYPE_V=f16
|
||||
|
||||
# MTP vs the drafter, MEASURED 2026-09-16 (temperature 0, 1k prompt / 512 out;
|
||||
# the autoregressive baseline and both MTP depths were run against the deployed
|
||||
# configuration in one session, same binary, same prompts):
|
||||
#
|
||||
# none 17.7 tok/s card1 26.07 GB
|
||||
# draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB
|
||||
# draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB
|
||||
# draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB
|
||||
#
|
||||
# All four produced IDENTICAL greedy text, so verification is lossless whichever
|
||||
# proposer runs. DFlash2 is worth 2.46x over autoregressive and 34 % over the
|
||||
# best MTP; deeper MTP drafting costs more than it returns (n=7 accepts 33 % of
|
||||
# 147 proposals, n=3 accepts 58 % of 72). The MTP head is not free either:
|
||||
# selecting it loads blk.64.nextn.* and adds ~3.3 GB to the card that holds it,
|
||||
# against 2.06 GB for the drafter's file. Do not switch this without
|
||||
# re-measuring.
|
||||
#
|
||||
# THE TWO CANNOT BE CHAINED. `--spec-type draft-mtp,draft-dflash` dies at load:
|
||||
# context type MTP requested but model doesn't contain MTP layers
|
||||
# failed to create MTP context
|
||||
# A chain has ONE draft context, and MTP builds against whatever model that
|
||||
# context holds -- here the DFlash2 file, which has no MTP layers. MTP's head is
|
||||
# only loaded when MTP IS the draft model ("creating MTP draft context against
|
||||
# the target model ..."), so a chain of the two is a structural dead end, not a
|
||||
# tuning question. See serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md.
|
||||
|
||||
# Text only: the server must not autoload a vision tower. Vision stays on
|
||||
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
|
||||
NO_MMPROJ=1
|
||||
Reference in new issue
Block a user