dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Use conservative four-token MTP drafts

Browse Source
This commit is contained in:
Frank committed 2026-08-25 04:20:15 -07:00
1 parent 3cf6353050
commit f86ed70c71
3 files changed
+7 -8
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/eemin-rocmi4-mtp.md
+5 -6
View File
@@ -86,13 +86,13 @@ sha256sum -c checksums.sha256
## Launch Profile ## Launch Profile
The qualified MTP profile (from the model card and PR #94, verified on the The qualified MTP profile (from the model card and PR #94, verified on the
live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`. live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
```mermaid ```mermaid
graph TD graph TD
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0] A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all] B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
C --> D[strict Qwen MTP<br/>n_max 16 / p_min 0.60<br/>backend sampling] C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
D --> E[Sharp chat template<br/>reasoning deepseek] D --> E[Sharp chat template<br/>reasoning deepseek]
``` ```
@@ -102,7 +102,7 @@ Key flags (see `scripts/serve-frank-large.sh`):
--spec-type draft-mtp --spec-mtp-strict-qwen --spec-type draft-mtp --spec-mtp-strict-qwen
--spec-draft-device ROCm0 --spec-draft-ngl all --spec-draft-device ROCm0 --spec-draft-ngl all
--spec-draft-type-k f16 --spec-draft-type-v f16 --spec-draft-type-k f16 --spec-draft-type-v f16
--spec-draft-n-max 16 --spec-draft-n-min 0 --spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.60 --spec-draft-backend-sampling --spec-draft-p-min 0.60 --spec-draft-backend-sampling
``` ```
@@ -133,9 +133,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation.
## Tuning MTP ## Tuning MTP
MTP gains are content-dependent: predictable code, structured data, and lists MTP gains are content-dependent: predictable code, structured data, and lists
accept longer drafts than creative prose. The qualified start is `n16 / p0.60`. accept longer drafts than creative prose. The active conservative profile is
If acceptance or stability is poor on another workload, lower `n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
`--spec-draft-n-max` to 4 or 6 and remeasure.
## ubatch Benchmark ## ubatch Benchmark
scripts/frank-default-large.env
+1 -1
View File
@@ -55,7 +55,7 @@ CACHE_RAM=8192
# Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF, # Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF,
# so the draft runs with all layers on ROCm0 (no --model-draft). # so the draft runs with all layers on ROCm0 (no --model-draft).
MTP_N_MAX=16 MTP_N_MAX=4
MTP_N_MIN=0 MTP_N_MIN=0
MTP_P_MIN=0.60 MTP_P_MIN=0.60
SPEC_MTP_STRICT_QWEN=on SPEC_MTP_STRICT_QWEN=on
tests/frank_large_config.bats
+1 -1
View File
@@ -33,7 +33,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE" grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE"
grep -qF -- '--spec-draft-p-min "$MTP_P_MIN"' "$SERVE_FILE" grep -qF -- '--spec-draft-p-min "$MTP_P_MIN"' "$SERVE_FILE"
# qualified profile tunables are exposed in the env # qualified profile tunables are exposed in the env
grep -q '^MTP_N_MAX=16$' "$ENV_FILE" grep -q '^MTP_N_MAX=4$' "$ENV_FILE"
grep -q '^MTP_P_MIN=0.60$' "$ENV_FILE" grep -q '^MTP_P_MIN=0.60$' "$ENV_FILE"
grep -q '^SPEC_MTP_STRICT_QWEN=on$' "$ENV_FILE" grep -q '^SPEC_MTP_STRICT_QWEN=on$' "$ENV_FILE"
grep -q '^MTP_BACKEND_SAMPLING=on$' "$ENV_FILE" grep -q '^MTP_BACKEND_SAMPLING=on$' "$ENV_FILE"