Use conservative four-token MTP drafts

This commit is contained in:
Frank
2026-08-25 04:20:15 -07:00
parent 3cf6353050
commit f86ed70c71
3 changed files with 7 additions and 8 deletions
+5 -6
View File
@@ -86,13 +86,13 @@ sha256sum -c checksums.sha256
## Launch Profile
The qualified MTP profile (from the model card and PR #94, verified on the
live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`.
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
```mermaid
graph TD
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
C --> D[strict Qwen MTP<br/>n_max 16 / p_min 0.60<br/>backend sampling]
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
D --> E[Sharp chat template<br/>reasoning deepseek]
```
@@ -102,7 +102,7 @@ Key flags (see `scripts/serve-frank-large.sh`):
--spec-type draft-mtp --spec-mtp-strict-qwen
--spec-draft-device ROCm0 --spec-draft-ngl all
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-draft-n-max 16 --spec-draft-n-min 0
--spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
```
@@ -133,9 +133,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
## Tuning MTP
MTP gains are content-dependent: predictable code, structured data, and lists
accept longer drafts than creative prose. The qualified start is `n16 / p0.60`.
If acceptance or stability is poor on another workload, lower
`--spec-draft-n-max` to 4 or 6 and remeasure.
accept longer drafts than creative prose. The active conservative profile is
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
## ubatch Benchmark