diff --git a/docs/LLM/eemin-rocmi4-mtp.md b/docs/LLM/eemin-rocmi4-mtp.md index 715b910..16ae9dc 100644 --- a/docs/LLM/eemin-rocmi4-mtp.md +++ b/docs/LLM/eemin-rocmi4-mtp.md @@ -86,13 +86,13 @@ sha256sum -c checksums.sha256 ## Launch Profile The qualified MTP profile (from the model card and PR #94, verified on the -live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`. +live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`. ```mermaid graph TD A[llama-server charlie12345 ROCmFPX c49ebdb
ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj
-ngl 999 -dev ROCm0] B --> C[draft-mtp on ROCm0
--spec-draft-ngl all] - C --> D[strict Qwen MTP
n_max 16 / p_min 0.60
backend sampling] + C --> D[strict Qwen MTP
n_max 4 / p_min 0.60
backend sampling] D --> E[Sharp chat template
reasoning deepseek] ``` @@ -102,7 +102,7 @@ Key flags (see `scripts/serve-frank-large.sh`): --spec-type draft-mtp --spec-mtp-strict-qwen --spec-draft-device ROCm0 --spec-draft-ngl all --spec-draft-type-k f16 --spec-draft-type-v f16 ---spec-draft-n-max 16 --spec-draft-n-min 0 +--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.60 --spec-draft-backend-sampling ``` @@ -133,9 +133,8 @@ path's greedy decisions; it does not remove the W4A4 activation approximation. ## Tuning MTP MTP gains are content-dependent: predictable code, structured data, and lists -accept longer drafts than creative prose. The qualified start is `n16 / p0.60`. -If acceptance or stability is poor on another workload, lower -`--spec-draft-n-max` to 4 or 6 and remeasure. +accept longer drafts than creative prose. The active conservative profile is +`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`. ## ubatch Benchmark diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 51903b4..b2b2ce8 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -55,7 +55,7 @@ CACHE_RAM=8192 # Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF, # so the draft runs with all layers on ROCm0 (no --model-draft). -MTP_N_MAX=16 +MTP_N_MAX=4 MTP_N_MIN=0 MTP_P_MIN=0.60 SPEC_MTP_STRICT_QWEN=on diff --git a/tests/frank_large_config.bats b/tests/frank_large_config.bats index 551778e..da8439e 100644 --- a/tests/frank_large_config.bats +++ b/tests/frank_large_config.bats @@ -33,7 +33,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE" grep -qF -- '--spec-draft-p-min "$MTP_P_MIN"' "$SERVE_FILE" # qualified profile tunables are exposed in the env - grep -q '^MTP_N_MAX=16$' "$ENV_FILE" + grep -q '^MTP_N_MAX=4$' "$ENV_FILE" grep -q '^MTP_P_MIN=0.60$' "$ENV_FILE" grep -q '^SPEC_MTP_STRICT_QWEN=on$' "$ENV_FILE" grep -q '^MTP_BACKEND_SAMPLING=on$' "$ENV_FILE"