dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use conservative four-token MTP drafts

Browse Source
This commit is contained in:
Frank
2026-08-25 04:20:15 -07:00
parent 3cf6353050
commit f86ed70c71
3 changed files with 7 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/eemin-rocmi4-mtp.md
+5 -6
View File
@@ -86,13 +86,13 @@ sha256sum -c checksums.sha256
## Launch Profile
The qualified MTP profile (from the model card and PR #94, verified on the
live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`.
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
```mermaid
graph TD
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
C --> D[strict Qwen MTP<br/>n_max 16 / p_min 0.60<br/>backend sampling]
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
D --> E[Sharp chat template<br/>reasoning deepseek]
```
@@ -102,7 +102,7 @@ Key flags (see `scripts/serve-frank-large.sh`):
--spec-type draft-mtp --spec-mtp-strict-qwen
--spec-draft-device ROCm0 --spec-draft-ngl all
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-draft-n-max 16 --spec-draft-n-min 0
--spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
```
@@ -133,9 +133,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
## Tuning MTP
MTP gains are content-dependent: predictable code, structured data, and lists
accept longer drafts than creative prose. The qualified start is `n16 / p0.60`.
If acceptance or stability is poor on another workload, lower
`--spec-draft-n-max` to 4 or 6 and remeasure.
accept longer drafts than creative prose. The active conservative profile is
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
## ubatch Benchmark
scripts/frank-default-large.env
+1 -1
View File
@@ -55,7 +55,7 @@ CACHE_RAM=8192
# Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF,
# so the draft runs with all layers on ROCm0 (no --model-draft).
MTP_N_MAX=16
MTP_N_MAX=4
MTP_N_MIN=0
MTP_P_MIN=0.60
SPEC_MTP_STRICT_QWEN=on
tests/frank_large_config.bats
+1 -1
View File
@@ -33,7 +33,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE"
grep -qF -- '--spec-draft-p-min "$MTP_P_MIN"' "$SERVE_FILE"
# qualified profile tunables are exposed in the env
grep -q '^MTP_N_MAX=16$' "$ENV_FILE"
grep -q '^MTP_N_MAX=4$' "$ENV_FILE"
grep -q '^MTP_P_MIN=0.60$' "$ENV_FILE"
grep -q '^SPEC_MTP_STRICT_QWEN=on$' "$ENV_FILE"
grep -q '^MTP_BACKEND_SAMPLING=on$' "$ENV_FILE"