Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix

This page records the speculative-decoding benchmark and the production tuning for the unsloth Qwen3.8-27B UD-Q8_K_XL service on Dankclaw (frank-default-large).

Result (applied, 2026-08-15: DSpark)

Model: Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP).
Spec: DSpark sidecar draft magnitudedev/Qwen3.8-27B-DSpark-GGUF (Qwen3.8-27B-DSpark-Q8_0.gguf, 1.36B, block_size 7), --spec-type draft-dspark, --spec-draft-n-max 7 (clamps to block_size).
PARALLEL=1. Two concurrent 16k prose prompts crashed the ROCm backend with ROCm error: an illegal memory access during per-sequence block sampling. See the DSpark parallel incident below.
Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).

Result (superseded 2026-08-14: bundled MTP)

Model: Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP).
Spec: draft-mtp, --spec-draft-n-max 5, no --spec-draft-p-min.
Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop.

Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)

A two-stream parallel acceptance test crashed frank-default-large.

graph TD
    A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
    B --> C[status 6/ABRT core dump]
    C --> D[systemd auto-restart, 503 during reload]
    D --> E[pinned PARALLEL=1]

Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts started simultaneously on PARALLEL=2. The ROCm backend aborted at ggml-cuda.cu:106 with ROCm error: an illegal memory access was encountered, surfacing in hipStreamSynchronize inside common_speculative_impl_draft_dflash::draft (the class used by draft-dspark). systemd caught it (status 6/ABRT) and restarted the unit; in-flight streams saw Remote end closed connection without response until the ~reload finished.

Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The DFlash draft model uses one shared draft context (ctx_dft) across all parallel slots; under concurrent block drafting on ROCm it performs an illegal memory access (HIP stream sync). This is the same multi-slot class of bug that retired draft-dflash for premature EOS; here it is a hard ROCm crash instead of a bad sample.

Fix: pin PARALLEL=1 (single slot). With one slot, concurrent requests queue safely and each completes to max_tokens. Revisit after an upstream multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k prose generation was ~steady (measuring only the correct-size signal, not this page).

Parallel results by stream count (2026-08-15, correct-size tests)

Parallel	Two 16k prose prompts	Two 67k prose prompts
2	crashes (illegal mem access)	passes
1	queues, both complete	n/a

Note: the crash manifests on the smaller (16k) concurrent streams, not the larger 67k pair. Size is not the trigger; concurrent block drafting on the shared draft context is.

How MTP works on Qwen3.8

The unsloth UD-* GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft context from the target model. Do not pass --spec-draft-model.

The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (9b05354ec) with GGML_HIP + gfx1151 + Vulkan.

Benchmark Method

The matrix ran 54 configurations: 3 spec types (draft-mtp,ngram-mod, draft-mtp, ngram-mod) x --spec-draft-n-max 2..10 x --spec-draft-p-min {off, 0.8}. Each config loaded llama-server, then ran 3 generations. The tool is scripts/spec-matrix-bench.sh; analysis is scripts/analyze-spec-matrix.py.

Matrix Caveat: Benchmark Prompt

The matrix used a repetitive 1100-token prompt (scripts/bench_live.sh style). MTP drafts that repeated sentence almost perfectly, which inflated the numbers for long drafts. The matrix reported draft-mtp n_max=10 at ~30 t/s. A realistic story prompt gave very different results:

n_max	median t/s (realistic)
4	12.5
5	13.3 (peak)
6	10.6
7	10.1
10	8.6
none (baseline)	7.2

Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost.

Premature Stop Investigation

Two failure patterns appeared during early production tests:

Speed collapse. ubatch >= 512 reduced MTP acceptance to ~25% and generation to ~9 t/s. ubatch 128 with batch >= 2048 triggered a premature instant-EOS under 2 concurrent slots. ubatch 256 avoids both: ~30 t/s single, concurrent-safe.
Apparent premature-EOS. The repetitive benchmark prompt with temp 1.0 sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early.

Production Configuration

scripts/frank-default-large.env:

SPEC_TYPE=draft-dspark
DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
SPEC_DRAFT_N_MAX=7
SPEC_DRAFT_P_MIN=
BATCH_SIZE=8192
UBATCH_SIZE=256
CTX_SIZE=524288
PARALLEL=1   # DSpark multi-slot crashes on this box (2026-08-15)

Window: computed at 262144 per slot (the pi models.json contextWindow for frank-default-large). The unit reports n_ctx_slot = 262144; CTX_SIZE is a pool ceiling and per-slot stays 262144 regardless of the single slot.

5.7 KiB Raw Blame History