Use conservative four-token MTP drafts
This commit is contained in:
@@ -86,13 +86,13 @@ sha256sum -c checksums.sha256
|
||||
## Launch Profile
|
||||
|
||||
The qualified MTP profile (from the model card and PR #94, verified on the
|
||||
live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`.
|
||||
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
|
||||
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
|
||||
C --> D[strict Qwen MTP<br/>n_max 16 / p_min 0.60<br/>backend sampling]
|
||||
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
|
||||
D --> E[Sharp chat template<br/>reasoning deepseek]
|
||||
```
|
||||
|
||||
@@ -102,7 +102,7 @@ Key flags (see `scripts/serve-frank-large.sh`):
|
||||
--spec-type draft-mtp --spec-mtp-strict-qwen
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl all
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
--spec-draft-n-max 16 --spec-draft-n-min 0
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
|
||||
```
|
||||
|
||||
@@ -133,9 +133,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
## Tuning MTP
|
||||
|
||||
MTP gains are content-dependent: predictable code, structured data, and lists
|
||||
accept longer drafts than creative prose. The qualified start is `n16 / p0.60`.
|
||||
If acceptance or stability is poor on another workload, lower
|
||||
`--spec-draft-n-max` to 4 or 6 and remeasure.
|
||||
accept longer drafts than creative prose. The active conservative profile is
|
||||
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
|
||||
|
||||
## ubatch Benchmark
|
||||
|
||||
|
||||
Reference in New Issue
Block a user