dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod)

Browse Source 
Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF
Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model).

- serve-frank-large.sh: SPEC_ARGS array, draft model existence check,
  spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0)
- frank-default-large.env: MTP tunables section, fix 'no MTP' comment
- kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in
  serving profile, EasiiX + PR #27739 citations

Set SPEC_TYPE=none in the env to disable speculative decoding.
This commit is contained in:
Frank
2026-08-27 23:52:42 -07:00
parent cadc7f2768
commit fc75ca6ed8
3 changed files with 107 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/kingjones-strix-rocmfp4.md
+48 -2
View File
@@ -21,7 +21,8 @@ timestamp: 2026-08-27
`frank-default-large` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP).
A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head
loaded as a separate GGUF sidecar (see MTP Draft Head below).
## Model Architecture
@@ -168,7 +169,12 @@ llama-server \
--presence-penalty 1.5 \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--spec-type draft-mtp,ngram-mod \
--model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-draft-n-max 4 --spec-draft-p-min 0.75 \
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
--spec-draft-type-k f16 --spec-draft-type-v f16
```
Key choices:
@@ -179,6 +185,44 @@ Key choices:
- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
internally.
- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod
fallback. The `ngram-mod` combo is recommended by the HF card; the
prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s
([[kingjones-strix-mtp]]). Benchmark both before committing.
- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance
prose loses more to rejected drafts than it gains (per the HF card).
## MTP Draft Head
The MTP (multi-token-prediction) draft head from
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded
as a separate GGUF sidecar via `--model-draft`.
| Field | Value |
|-------|-------|
| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` |
| Size | 4.1 GB (4,137,429,088 B) |
| Quant | Q8_0 |
| Architecture | `qwen4exp` |
| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF |
**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on
code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth.
Speculative decoding is lossless at temp 0. Our service runs at temp 0.7,
so expect lower acceptance; the `p_min 0.75` gate compensates.
**Caveats:**
- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is
the "Uncensored" (abliterated) variant. Abliteration modifies a small
number of weight vectors, not the full model — acceptance may be
marginally lower.
- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our
`charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp)
merged, so the MTP mechanism is the same upstream code. If the draft
load crashes, check for Strix-Halo kernel patch differences first.
- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable.
## Deployment
@@ -228,3 +272,5 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support