frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod)
Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model). - serve-frank-large.sh: SPEC_ARGS array, draft model existence check, spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0) - frank-default-large.env: MTP tunables section, fix 'no MTP' comment - kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in serving profile, EasiiX + PR #27739 citations Set SPEC_TYPE=none in the env to disable speculative decoding.
This commit is contained in:
1 parent
cadc7f2768
commit
fc75ca6ed8
3 files changed
+107
-5
No files matched your search
@@ -17,7 +17,8 @@
|
||||
# (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
|
||||
# Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
|
||||
# PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
|
||||
# Text-only (no mmproj, no MTP draft head).
|
||||
# Text-only (no mmproj). MTP draft head loaded as a separate GGUF sidecar
|
||||
# (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, Q8_0, 4.1 GB).
|
||||
#
|
||||
# Context: 262144 (256k) x 2 slots. f16 KV cache (~6.4 GiB per slot,
|
||||
# ~12.8 GiB total). --fit off because the auto-fit code segfaults on this
|
||||
@@ -82,4 +83,20 @@ REASONING_FORMAT=deepseek
|
||||
# Strix Halo unified memory + native gfx1151.
|
||||
HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
|
||||
VERBOSITY=4
|
||||
VERBOSITY=4
|
||||
|
||||
# MTP draft head (speculative decoding). The MTP head from
|
||||
# Qwen/Qwen3.8-Flash-Next, exported as a standalone Q8_0 GGUF sidecar by
|
||||
# EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF. Architecture qwen4exp,
|
||||
# matching the main model. Measured +50% decode on code at temp 0 on
|
||||
# Strix Halo (gfx1151). Set SPEC_TYPE=none to disable.
|
||||
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next
|
||||
DRAFT_MODEL_FILE=mtp-Qwen3.8-Flash-Next-Q8_0.gguf
|
||||
SPEC_TYPE=draft-mtp,ngram-mod
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.75
|
||||
SPEC_DRAFT_NGL=999
|
||||
SPEC_DRAFT_DEVICE=ROCm0
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
SPEC_DRAFT_TYPE_V=f16
|
||||
Reference in new issue
Block a user