From fc75ca6ed80ab225e3da58aca0bc4138d68ef9b6 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 27 Aug 2026 23:52:42 -0700 Subject: [PATCH] frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod) Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model). - serve-frank-large.sh: SPEC_ARGS array, draft model existence check, spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0) - frank-default-large.env: MTP tunables section, fix 'no MTP' comment - kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in serving profile, EasiiX + PR #27739 citations Set SPEC_TYPE=none in the env to disable speculative decoding. --- docs/LLM/kingjones-strix-rocmfp4.md | 50 +++++++++++++++++++++++++++-- scripts/frank-default-large.env | 21 ++++++++++-- scripts/serve-frank-large.sh | 41 ++++++++++++++++++++++- 3 files changed, 107 insertions(+), 5 deletions(-) diff --git a/docs/LLM/kingjones-strix-rocmfp4.md b/docs/LLM/kingjones-strix-rocmfp4.md index 902c221..b4284d7 100644 --- a/docs/LLM/kingjones-strix-rocmfp4.md +++ b/docs/LLM/kingjones-strix-rocmfp4.md @@ -21,7 +21,8 @@ timestamp: 2026-08-27 `frank-default-large` (port 7770) serves `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face. -A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP). +A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head +loaded as a separate GGUF sidecar (see MTP Draft Head below). ## Model Architecture @@ -168,7 +169,12 @@ llama-server \ --presence-penalty 1.5 \ --reasoning on --reasoning-format deepseek --reasoning-budget -1 \ --chat-template-file chat-template.jinja \ - --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' + --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \ + --spec-type draft-mtp,ngram-mod \ + --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \ + --spec-draft-n-max 4 --spec-draft-p-min 0.75 \ + --spec-draft-ngl 999 --spec-draft-device ROCm0 \ + --spec-draft-type-k f16 --spec-draft-type-v f16 ``` Key choices: @@ -179,6 +185,44 @@ Key choices: - **`--gpu-layers 999`** — all layers on SVM-backed GPU pool. - **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps internally. +- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod + fallback. The `ngram-mod` combo is recommended by the HF card; the + prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s + ([[kingjones-strix-mtp]]). Benchmark both before committing. +- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance + prose loses more to rejected drafts than it gains (per the HF card). + +## MTP Draft Head + +The MTP (multi-token-prediction) draft head from +[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) +(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded +as a separate GGUF sidecar via `--model-draft`. + +| Field | Value | +|-------|-------| +| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` | +| Size | 4.1 GB (4,137,429,088 B) | +| Quant | Q8_0 | +| Architecture | `qwen4exp` | +| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF | + +**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on +code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth. +Speculative decoding is lossless at temp 0. Our service runs at temp 0.7, +so expect lower acceptance; the `p_min 0.75` gate compensates. + +**Caveats:** + +- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is + the "Uncensored" (abliterated) variant. Abliteration modifies a small + number of weight vectors, not the full model — acceptance may be + marginally lower. +- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our + `charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp) + merged, so the MTP mechanism is the same upstream code. If the draft + load crashes, check for Strix-Halo kernel patch differences first. +- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable. ## Deployment @@ -228,3 +272,5 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network. - [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF) - [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) - [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) +- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar +- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 60c0fd8..a1389f5 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -17,7 +17,8 @@ # (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8). # Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file. # PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM). -# Text-only (no mmproj, no MTP draft head). +# Text-only (no mmproj). MTP draft head loaded as a separate GGUF sidecar +# (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, Q8_0, 4.1 GB). # # Context: 262144 (256k) x 2 slots. f16 KV cache (~6.4 GiB per slot, # ~12.8 GiB total). --fit off because the auto-fit code segfaults on this @@ -82,4 +83,20 @@ REASONING_FORMAT=deepseek # Strix Halo unified memory + native gfx1151. HSA_OVERRIDE_GFX_VERSION=11.5.1 -VERBOSITY=4 \ No newline at end of file +VERBOSITY=4 + +# MTP draft head (speculative decoding). The MTP head from +# Qwen/Qwen3.8-Flash-Next, exported as a standalone Q8_0 GGUF sidecar by +# EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF. Architecture qwen4exp, +# matching the main model. Measured +50% decode on code at temp 0 on +# Strix Halo (gfx1151). Set SPEC_TYPE=none to disable. +DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next +DRAFT_MODEL_FILE=mtp-Qwen3.8-Flash-Next-Q8_0.gguf +SPEC_TYPE=draft-mtp,ngram-mod +SPEC_DRAFT_N_MAX=4 +SPEC_DRAFT_N_MIN=0 +SPEC_DRAFT_P_MIN=0.75 +SPEC_DRAFT_NGL=999 +SPEC_DRAFT_DEVICE=ROCm0 +SPEC_DRAFT_TYPE_K=f16 +SPEC_DRAFT_TYPE_V=f16 \ No newline at end of file diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 9c7747d..62efeea 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -21,6 +21,11 @@ # hyper-connection/indexer configuration; -ngl 999 places all layers on # the GPU (SVM-backed). # +# MTP draft head: EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF (Q8_0, +# 4.1 GB). Architecture qwen4exp, matching the main model. Speculative +# decoding via --spec-type draft-mtp,ngram-mod (n_max 4, p_min 0.75). +# Set SPEC_TYPE=none in the env to disable. +# # The Qwen-Sharp chat template (scripts/chat-template.jinja) is applied at # runtime so the model emits reasoning in DeepSeek format; the reasoning # trail is preserved across turns via CHAT_TEMPLATE_KWARGS. @@ -63,6 +68,16 @@ fi : "${REASONING:=on}" : "${REASONING_FORMAT:=deepseek}" : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" +: "${SPEC_TYPE:=draft-mtp,ngram-mod}" +: "${DRAFT_MODEL_DIR:=$MODEL_DIR}" +: "${DRAFT_MODEL_FILE:=mtp-Qwen3.8-Flash-Next-Q8_0.gguf}" +: "${SPEC_DRAFT_N_MAX:=4}" +: "${SPEC_DRAFT_N_MIN:=0}" +: "${SPEC_DRAFT_P_MIN:=0.75}" +: "${SPEC_DRAFT_NGL:=999}" +: "${SPEC_DRAFT_DEVICE:=ROCm0}" +: "${SPEC_DRAFT_TYPE_K:=f16}" +: "${SPEC_DRAFT_TYPE_V:=f16}" MODEL="$MODEL_DIR/$MODEL_FILE" for f in "$LLAMA_BIN" "$MODEL" "$CHAT_TEMPLATE_FILE"; do @@ -76,6 +91,29 @@ if [[ ! -x "$LLAMA_BIN" ]]; then exit 1 fi +# --- speculative decoding (MTP draft head) -------------------------------- +# The MTP head is loaded as a separate GGUF sidecar via --model-draft. +# Set SPEC_TYPE=none in the env to disable speculative decoding entirely. +SPEC_ARGS=() +if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then + DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE" + if [[ ! -r "$DRAFT_MODEL" ]]; then + echo "missing or unreadable draft model: $DRAFT_MODEL" >&2 + exit 1 + fi + SPEC_ARGS=( + --spec-type "$SPEC_TYPE" + --model-draft "$DRAFT_MODEL" + --spec-draft-n-max "$SPEC_DRAFT_N_MAX" + --spec-draft-n-min "$SPEC_DRAFT_N_MIN" + --spec-draft-p-min "$SPEC_DRAFT_P_MIN" + --spec-draft-ngl "$SPEC_DRAFT_NGL" + --spec-draft-device "$SPEC_DRAFT_DEVICE" + --spec-draft-type-k "$SPEC_DRAFT_TYPE_K" + --spec-draft-type-v "$SPEC_DRAFT_TYPE_V" + ) +fi + SERVER_DIR="$(cd -- "$(dirname -- "$LLAMA_BIN")" && pwd)" ROCM_PATH="${ROCM_PATH:-/opt/rocm}" LIBRARY_PATH="$SERVER_DIR:$ROCM_PATH/lib:$ROCM_PATH/lib/rocm_sysdeps/lib:$ROCM_PATH/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" @@ -117,4 +155,5 @@ exec /usr/bin/env \ --presence-penalty "$PRESENCE_PENALTY" \ --reasoning "$REASONING" \ --reasoning-format "$REASONING_FORMAT" \ - --reasoning-budget -1 \ No newline at end of file + --reasoning-budget -1 \ + "${SPEC_ARGS[@]}" \ No newline at end of file