frank-medium: the drafter beats MTP and MTP beats nothing -- measured, not assumed
Four configurations of the same binary, identical prompts at temperature 0, the unit stopped between runs, on the deployed payload: none 17.7 tok/s card1 26.07 GB draft-mtp n=3 32.4 tok/s 58 % accepted card1 29.35 GB draft-mtp n=7 26.7 tok/s 33 % accepted card1 29.65 GB draft-dflash 7 43.5 tok/s 41 % accepted card1 29.42 GB DFlash2 is worth 2.46x over autoregressive and 34 % over the best MTP, and all four produced IDENTICAL greedy text, so the 'lossless verification' claim in the runbook is now a check rather than an assumption. MTP is not the cheaper option either: its head loads ~3.3 GB against the drafter's 2.06 GB file, and deeper drafting loses (n=7 accepts a third of its proposals, n=3 more than half). The two CANNOT be chained, and the failure is structural rather than tunable: 'spec-type draft-mtp,draft-dflash' dies at load with 'context type MTP requested but model doesn't contain MTP layers', because a chain has ONE draft context and MTP builds against whatever model that context holds -- here the DFlash2 file. The serve script therefore passes --spec-draft-model in the dflash branch ONLY, and the seam now asserts exactly that (the wiring whose violation is this error), plus that the deployment stays on the measured winner. scripts/bench-spec-ab.py makes the comparison reproducible: the benchmarking policy wants the runtime's own harness for this, and the numbers above were otherwise unreproducible one-offs.
This commit is contained in:
1 parent
9a83dae73a
commit
34253a9903
4 files changed
+269
No files matched your search
@@ -73,6 +73,26 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter
|
||||
[ -n "$n_max" ] && [ "$n_max" -ge 1 ] && [ "$n_max" -le 7 ]
|
||||
}
|
||||
|
||||
@test "a draft MODEL is passed only for DFlash types, never for MTP" {
|
||||
# MTP builds its context against the TARGET model (its head lives in that
|
||||
# GGUF). Passing --spec-draft-model alongside it points the draft context at
|
||||
# the drafter's file, which has no MTP layers, and the server dies at load:
|
||||
# context type MTP requested but model doesn't contain MTP layers
|
||||
# failed to create MTP context
|
||||
# So the flag must appear exactly once, inside the dflash branch.
|
||||
python3 -c "
|
||||
lines = open('$SERVE').read().splitlines()
|
||||
guard = next(i for i, l in enumerate(lines) if '*dflash*' in l)
|
||||
indent = len(lines[guard]) - len(lines[guard].lstrip())
|
||||
end = next(i for i in range(guard + 1, len(lines))
|
||||
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
|
||||
block = chr(10).join(lines[guard:end])
|
||||
assert '--spec-draft-model' in block, 'the dflash branch must pass the draft model'
|
||||
assert sum(l.count('--spec-draft-model') for l in lines) == 1, 'only that branch may pass it'
|
||||
print('draft-model wiring ok')
|
||||
"
|
||||
}
|
||||
|
||||
@test "the runtime is pinned to a commit and to gfx1201 alone" {
|
||||
# A floating ref would silently move the service between engine versions,
|
||||
# and an unpinned arch would build for a sibling GPU that loads and then
|
||||
|
||||
Reference in new issue
Block a user