frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod)

Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF
Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model).

- serve-frank-large.sh: SPEC_ARGS array, draft model existence check,
  spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0)
- frank-default-large.env: MTP tunables section, fix 'no MTP' comment
- kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in
  serving profile, EasiiX + PR #27739 citations

Set SPEC_TYPE=none in the env to disable speculative decoding.
This commit is contained in:
Frank committed 2026-08-27 23:52:42 -07:00
1 parent cadc7f2768
commit fc75ca6ed8
3 files changed
+107 -5

No files matched your search

+19 -2
View File
@@ -17,7 +17,8 @@
# (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
# Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
# PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
# Text-only (no mmproj, no MTP draft head).
# Text-only (no mmproj). MTP draft head loaded as a separate GGUF sidecar
# (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, Q8_0, 4.1 GB).
#
# Context: 262144 (256k) x 2 slots. f16 KV cache (~6.4 GiB per slot,
# ~12.8 GiB total). --fit off because the auto-fit code segfaults on this
@@ -82,4 +83,20 @@ REASONING_FORMAT=deepseek
# Strix Halo unified memory + native gfx1151.
HSA_OVERRIDE_GFX_VERSION=11.5.1
VERBOSITY=4
VERBOSITY=4
# MTP draft head (speculative decoding). The MTP head from
# Qwen/Qwen3.8-Flash-Next, exported as a standalone Q8_0 GGUF sidecar by
# EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF. Architecture qwen4exp,
# matching the main model. Measured +50% decode on code at temp 0 on
# Strix Halo (gfx1151). Set SPEC_TYPE=none to disable.
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next
DRAFT_MODEL_FILE=mtp-Qwen3.8-Flash-Next-Q8_0.gguf
SPEC_TYPE=draft-mtp,ngram-mod
SPEC_DRAFT_N_MAX=4
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.75
SPEC_DRAFT_NGL=999
SPEC_DRAFT_DEVICE=ROCm0
SPEC_DRAFT_TYPE_K=f16
SPEC_DRAFT_TYPE_V=f16