serverAppConfigs/docs/DFlash DSpark Speculative Decoding.md

DFlash and DSpark Speculative Decoding

Overview

DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.

DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.

How DFlash Works

DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:

• The draft model reuses the embedding and LM head from the target model.
• Only the intermediate layers are trained.
• It fuses target-model hidden features into the KV projections of every draft layer.
• A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.

Results from the DFlash paper (arXiv:2602.06036):

• Over 6x lossless speedup on Qwen3-8B.
• Up to 2.5x higher speedup than EAGLE-3.
• Diffusion drafting cost is flat, so block size does not increase cost.

How DSpark Works

DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.

Results from the DSpark paper:

• 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
• Useful when parallel drafts decay in acceptance rate.

Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.

Runtime Support

Runtime	DFlash	DSpark	MTP
vLLM	yes (v0.20.1+)	yes	yes
llama.cpp	yes	yes	yes
SGLang	yes	no	yes

vLLM

vLLM v0.20.1+ has core DFlash support. Use the --speculative-config JSON object.

DFlash on vLLM:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
    --attention-backend flash_attn \
    --max-num-batched-tokens 32768

Legacy flags also work:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --spec-method dflash \
    --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
    --spec-tokens 8

DFlash requires a non-causal attention backend. Set "attention_backend": "flash_attn" in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.

DSpark on vLLM:

vllm serve <target-model> \
    --speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'

llama.cpp

llama.cpp supports DFlash, DSpark, and MTP through --spec-type. When you pass --spec-draft-hf, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.

--spec-type accepts a comma-separated list of speculative implementations. common_speculative_init (in common/speculative.cpp) builds a vector of implementations and chains drafting through all of them (common_speculative_draft_params.drafting is "used to chain the drafts through all the available implementations"). This means you can layer ngram-mod on top of a DFlash draft model in one server process:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <dflash-draft-GGUF> \
    --spec-type draft-dflash,ngram-mod

The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream common/speculative.cpp (b10133+).

DFlash on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dflash \
    --spec-draft-n-max 16

DSpark on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dspark

MTP on llama.cpp:

llama-server --hf-repo <target-with-MTP> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2

Note: the flag was renamed from --spec-type mtp to --spec-type draft-mtp in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.

MTP drafter availability for Qwen3.6-27B

MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's draft-mtp implementation is fully wired for the qwen35 and qwen35moe architectures (PR #22673).

Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:

File	Architecture	Tensors	MTP tensors
Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf	qwen35	851	0
Qwen3.6-27B-DFlash-Q8_0.gguf (draft)	dflash	58	0

The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:

llama-server --model <HauhauCS-Q8_K_P.gguf> \
    --spec-draft-model <MTP-only-draft.gguf> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2

Compatible MTP-only draft GGUFs on Hugging Face:

Repo	Notes
a4lg/Qwen3.6-27B-MTP-ONLY-GGUF	MTP-only subset of the base model; pairs with derivative/abliterated targets via --spec-draft-model
lym00/Qwen3.6-27B-MTP-ONLY-GGUF	Same purpose; built from convert_hf_to_gguf.py --mtp
ggml-org/Qwen3.6-27B-MTP-GGUF	Full model with MTP head (replaces the target, not a sidecar)
hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF	HauhauCS Q4_K_P with the MTP head grafted in (replaces the target)

Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.

On Dankclaw, draft-dflash remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see deployment).

Draft Models

DFlash draft model sizes for the models on these servers:

Target model	DFlash draft	Draft size
Qwen3.6-35B-A3B	z-lab/Qwen3.6-35B-A3B-DFlash	0.4B
Qwen3.6-27B	z-lab/Qwen3.6-27B-DFlash	2B
Qwen3.5-35B-A3B	z-lab/Qwen3.5-35B-A3B-DFlash	0.4B
Qwen3.5-27B	z-lab/Qwen3.5-27B-DFlash	2B

The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.

Deployment on Dankserver (NVIDIA RTX 3090)

Deployment on Dankserver (NVIDIA RTX 3090)

The active Dankserver deployment runs llama.cpp (frank-default-small), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P on port 7770 using ngram-mod speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See deployment and scripts/serve-llama-frank-small.sh.

DFlash retired from active service. DFlash (draft-dflash) was tested initially but did not deliver the desired speed gains under the frank-default-small workload (see ngram-mod benchmarks below). The Alittlehammmer/Qwen3.6-35B-A3B-DFlash draft GGUF remains in /home/dank/models/ as reference material only.

The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:

• FP8 KV cache is valid on CUDA 12.
• FP8 weights compute through a fallback path; no native FP8 tensor cores.
• The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.

Working DFlash flags for this server:

--spec-method dflash
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
--spec-tokens 8
--kv-cache-dtype fp8

Example DFlash + AWQ on the RTX 3090:

docker run --rm --name vllm-qwen36-dflash \
    --gpus all --ipc=host \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    docker.io/vllm/vllm-openai:latest \
    cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
    --attention-backend flash_attn \
    --gpu-memory-utilization 0.95

Deployment on Dankclaw (AMD Strix Halo gfx1151)

Dankclaw now runs llama.cpp as its runtime for frank-default-large (port 7770), serving the HauhauCS Qwen3.6-27B uncensored Q8_K_P model with a DFlash draft model (draft-dflash) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.

Current facts:

• Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive Q8_K_P (dense 27B).
• Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp Q8_0 (block size 16).
• Spec command: --spec-type draft-dflash with --spec-draft-n-max 8.
• Backend --device ROCm0; --gpu-layers all --fit off; KV q8_0 K / f16 V.
• Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.

Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See frank-default-large-benchmark for full numbers and the memory-bound batch/ubatch limits.

Notes

• Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
• DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
• The old --swap-space flag no longer exists in current vLLM. Use --kv-offloading-size.
• --speculative-config is the preferred CLI. Legacy --spec-method, --spec-model, and --spec-tokens flags still exist.
• Use rejection_sample_method default strict for lossless output.

See Also

• Quantization by Hardware Architecture
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw
• Dankclaw System Specs
• Dankserver System Specs
• VLLM Overview
• deployment