Frank
0eda208705
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not 10), the ubatch findings (256 is the safe/fast point), and the premature-stop sampling-variance investigation.
2.8 KiB
Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
This page records the speculative-decoding benchmark and the production
tuning for the unsloth Qwen3.8-27B UD-Q8_K_XL service on Dankclaw
(frank-default-large).
Result (applied)
- Model:
Qwen3.8-27B-UD-Q8_K_XL.gguf(29.29 GiB, bundled MTP). - Spec:
draft-mtp,--spec-draft-n-max 5, no--spec-draft-p-min. - Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop.
How MTP works on Qwen3.8
The unsloth UD-* GGUFs keep the MTP (Multi-Token Prediction) head inside
the model file. llama.cpp b10411+ auto-detects it and creates the draft
context from the target model. Do not pass --spec-draft-model.
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
on Dankclaw is b10433 (9b05354ec) with GGML_HIP + gfx1151 + Vulkan.
Benchmark Method
The matrix ran 54 configurations: 3 spec types (draft-mtp,ngram-mod,
draft-mtp, ngram-mod) x --spec-draft-n-max 2..10 x
--spec-draft-p-min {off, 0.8}. Each config loaded llama-server,
then ran 3 generations. The tool is scripts/spec-matrix-bench.sh;
analysis is scripts/analyze-spec-matrix.py.
Matrix Caveat: Benchmark Prompt
The matrix used a repetitive 1100-token prompt
(scripts/bench_live.sh style). MTP drafts that repeated sentence almost
perfectly, which inflated the numbers for long drafts. The matrix reported
draft-mtp n_max=10 at ~30 t/s. A realistic story prompt gave very
different results:
| n_max | median t/s (realistic) |
|---|---|
| 4 | 12.5 |
| 5 | 13.3 (peak) |
| 6 | 10.6 |
| 7 | 10.1 |
| 10 | 8.6 |
| none (baseline) | 7.2 |
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost.
Premature Stop Investigation
Two failure patterns appeared during early production tests:
- Speed collapse.
ubatch >= 512reduced MTP acceptance to ~25% and generation to ~9 t/s.ubatch 128withbatch >= 2048triggered a premature instant-EOS under 2 concurrent slots.ubatch 256avoids both: ~30 t/s single, concurrent-safe. - Apparent premature-EOS. The repetitive benchmark prompt with
temp 1.0sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early.
Production Configuration
scripts/frank-default-large.env:
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=5
SPEC_DRAFT_P_MIN=
BATCH_SIZE=8192
UBATCH_SIZE=256
CTX_SIZE=524288
PARALLEL=2
Window: 524288 total / 2 = 262144 per slot, matching the pi
models.json contextWindow for frank-default-large.