frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod)

Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model). - serve-frank-large.sh: SPEC_ARGS array, draft model existence check, spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0) - frank-default-large.env: MTP tunables section, fix 'no MTP' comment - kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in serving profile, EasiiX + PR #27739 citations Set SPEC_TYPE=none in the env to disable speculative decoding.
2026-08-27 23:52:42 -07:00
parent cadc7f2768
commit fc75ca6ed8
3 changed files with 107 additions and 5 deletions
@@ -21,7 +21,8 @@ timestamp: 2026-08-27

 `frank-default-large` (port 7770) serves
 `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
-A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP).
+A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head
+loaded as a separate GGUF sidecar (see MTP Draft Head below).

 ## Model Architecture

@@ -168,7 +169,12 @@ llama-server \
  --presence-penalty 1.5 \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
-  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
+  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
+  --spec-type draft-mtp,ngram-mod \
+  --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
+  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
+  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
+  --spec-draft-type-k f16 --spec-draft-type-v f16
 ```

 Key choices:
@@ -179,6 +185,44 @@ Key choices:
 - **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
 - **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
  internally.
+- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod
+  fallback. The `ngram-mod` combo is recommended by the HF card; the
+  prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s
+  ([[kingjones-strix-mtp]]). Benchmark both before committing.
+- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance
+  prose loses more to rejected drafts than it gains (per the HF card).
+
+## MTP Draft Head
+
+The MTP (multi-token-prediction) draft head from
+[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
+(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded
+as a separate GGUF sidecar via `--model-draft`.
+
+| Field | Value |
+|-------|-------|
+| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` |
+| Size | 4.1 GB (4,137,429,088 B) |
+| Quant | Q8_0 |
+| Architecture | `qwen4exp` |
+| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF |
+
+**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on
+code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth.
+Speculative decoding is lossless at temp 0. Our service runs at temp 0.7,
+so expect lower acceptance; the `p_min 0.75` gate compensates.
+
+**Caveats:**
+
+- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is
+  the "Uncensored" (abliterated) variant. Abliteration modifies a small
+  number of weight vectors, not the full model — acceptance may be
+  marginally lower.
+- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our
+  `charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp)
+  merged, so the MTP mechanism is the same upstream code. If the draft
+  load crashes, check for Strix-Halo kernel patch differences first.
+- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable.

 ## Deployment

@@ -228,3 +272,5 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
 - [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
 - [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
 - [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
+- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar
+- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
@@ -17,7 +17,8 @@
 # (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
 # Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
 # PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
-# Text-only (no mmproj, no MTP draft head).
+# Text-only (no mmproj).  MTP draft head loaded as a separate GGUF sidecar
+# (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, Q8_0, 4.1 GB).
 #
 # Context: 262144 (256k) x 2 slots.  f16 KV cache (~6.4 GiB per slot,
 # ~12.8 GiB total).  --fit off because the auto-fit code segfaults on this
@@ -82,4 +83,20 @@ REASONING_FORMAT=deepseek
 # Strix Halo unified memory + native gfx1151.
 HSA_OVERRIDE_GFX_VERSION=11.5.1

-VERBOSITY=4
+VERBOSITY=4
+
+# MTP draft head (speculative decoding).  The MTP head from
+# Qwen/Qwen3.8-Flash-Next, exported as a standalone Q8_0 GGUF sidecar by
+# EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF.  Architecture qwen4exp,
+# matching the main model.  Measured +50% decode on code at temp 0 on
+# Strix Halo (gfx1151).  Set SPEC_TYPE=none to disable.
+DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next
+DRAFT_MODEL_FILE=mtp-Qwen3.8-Flash-Next-Q8_0.gguf
+SPEC_TYPE=draft-mtp,ngram-mod
+SPEC_DRAFT_N_MAX=4
+SPEC_DRAFT_N_MIN=0
+SPEC_DRAFT_P_MIN=0.75
+SPEC_DRAFT_NGL=999
+SPEC_DRAFT_DEVICE=ROCm0
+SPEC_DRAFT_TYPE_K=f16
+SPEC_DRAFT_TYPE_V=f16
@@ -21,6 +21,11 @@
 # hyper-connection/indexer configuration; -ngl 999 places all layers on
 # the GPU (SVM-backed).
 #
+# MTP draft head: EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF (Q8_0,
+# 4.1 GB).  Architecture qwen4exp, matching the main model.  Speculative
+# decoding via --spec-type draft-mtp,ngram-mod (n_max 4, p_min 0.75).
+# Set SPEC_TYPE=none in the env to disable.
+#
 # The Qwen-Sharp chat template (scripts/chat-template.jinja) is applied at
 # runtime so the model emits reasoning in DeepSeek format; the reasoning
 # trail is preserved across turns via CHAT_TEMPLATE_KWARGS.
@@ -63,6 +68,16 @@ fi
 : "${REASONING:=on}"
 : "${REASONING_FORMAT:=deepseek}"
 : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
+: "${SPEC_TYPE:=draft-mtp,ngram-mod}"
+: "${DRAFT_MODEL_DIR:=$MODEL_DIR}"
+: "${DRAFT_MODEL_FILE:=mtp-Qwen3.8-Flash-Next-Q8_0.gguf}"
+: "${SPEC_DRAFT_N_MAX:=4}"
+: "${SPEC_DRAFT_N_MIN:=0}"
+: "${SPEC_DRAFT_P_MIN:=0.75}"
+: "${SPEC_DRAFT_NGL:=999}"
+: "${SPEC_DRAFT_DEVICE:=ROCm0}"
+: "${SPEC_DRAFT_TYPE_K:=f16}"
+: "${SPEC_DRAFT_TYPE_V:=f16}"

 MODEL="$MODEL_DIR/$MODEL_FILE"
 for f in "$LLAMA_BIN" "$MODEL" "$CHAT_TEMPLATE_FILE"; do
@@ -76,6 +91,29 @@ if [[ ! -x "$LLAMA_BIN" ]]; then
    exit 1
 fi

+# --- speculative decoding (MTP draft head) --------------------------------
+# The MTP head is loaded as a separate GGUF sidecar via --model-draft.
+# Set SPEC_TYPE=none in the env to disable speculative decoding entirely.
+SPEC_ARGS=()
+if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
+    DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
+    if [[ ! -r "$DRAFT_MODEL" ]]; then
+        echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
+        exit 1
+    fi
+    SPEC_ARGS=(
+        --spec-type "$SPEC_TYPE"
+        --model-draft "$DRAFT_MODEL"
+        --spec-draft-n-max "$SPEC_DRAFT_N_MAX"
+        --spec-draft-n-min "$SPEC_DRAFT_N_MIN"
+        --spec-draft-p-min "$SPEC_DRAFT_P_MIN"
+        --spec-draft-ngl "$SPEC_DRAFT_NGL"
+        --spec-draft-device "$SPEC_DRAFT_DEVICE"
+        --spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
+        --spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
+    )
+fi
+
 SERVER_DIR="$(cd -- "$(dirname -- "$LLAMA_BIN")" && pwd)"
 ROCM_PATH="${ROCM_PATH:-/opt/rocm}"
 LIBRARY_PATH="$SERVER_DIR:$ROCM_PATH/lib:$ROCM_PATH/lib/rocm_sysdeps/lib:$ROCM_PATH/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
@@ -117,4 +155,5 @@ exec /usr/bin/env \
    --presence-penalty "$PRESENCE_PENALTY" \
    --reasoning "$REASONING" \
    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget -1
+    --reasoning-budget -1 \
+    "${SPEC_ARGS[@]}"