serverAppConfigs/docs/DFlash DSpark Speculative Decoding.md

DFlash and DSpark Speculative Decoding

Overview

DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.

DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.

How DFlash Works

DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:

• The draft model reuses the embedding and LM head from the target model.
• Only the intermediate layers are trained.
• It fuses target-model hidden features into the KV projections of every draft layer.
• A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.

Results from the DFlash paper (arXiv:2602.06036):

• Over 6x lossless speedup on Qwen3-8B.
• Up to 2.5x higher speedup than EAGLE-3.
• Diffusion drafting cost is flat, so block size does not increase cost.

How DSpark Works

DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.

Results from the DSpark paper:

• 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
• Useful when parallel drafts decay in acceptance rate.

Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.

Runtime Support

Runtime	DFlash	DSpark	MTP
vLLM	yes (v0.20.1+)	yes	yes
llama.cpp	yes	yes	yes
SGLang	yes	no	yes

vLLM

vLLM v0.20.1+ has core DFlash support. Use the --speculative-config JSON object.

DFlash on vLLM:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
    --attention-backend flash_attn \
    --max-num-batched-tokens 32768

Legacy flags also work:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --spec-method dflash \
    --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
    --spec-tokens 8

DFlash requires a non-causal attention backend. Set "attention_backend": "flash_attn" in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.

DSpark on vLLM:

vllm serve <target-model> \
    --speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'

llama.cpp

llama.cpp supports DFlash, DSpark, and MTP through --spec-type. When you pass --spec-draft-hf, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.

--spec-type accepts a comma-separated list of speculative implementations. common_speculative_init (in common/speculative.cpp) builds a vector of implementations and chains drafting through all of them (common_speculative_draft_params.drafting is "used to chain the drafts through all the available implementations"). This means you can layer ngram-mod on top of a DFlash draft model in one server process:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <dflash-draft-GGUF> \
    --spec-type draft-dflash,ngram-mod

The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream common/speculative.cpp (b10133+).

DFlash on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dflash \
    --spec-draft-n-max 16

DSpark on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dspark

MTP on llama.cpp:

llama-server --hf-repo <target-with-MTP> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2

Draft Models

DFlash draft model sizes for the models on these servers:

Target model	DFlash draft	Draft size
Qwen3.6-35B-A3B	z-lab/Qwen3.6-35B-A3B-DFlash	0.4B
Qwen3.6-27B	z-lab/Qwen3.6-27B-DFlash	2B
Qwen3.5-35B-A3B	z-lab/Qwen3.5-35B-A3B-DFlash	0.4B
Qwen3.5-27B	z-lab/Qwen3.5-27B-DFlash	2B

The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.

Deployment on Dankserver (NVIDIA RTX 3090)

Deployment on Dankserver (NVIDIA RTX 3090)

The active Dankserver deployment runs llama.cpp (frank-default-small), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored Q6_K_P on port 7770 using ngram-mod speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See deployment and scripts/serve-llama-frank-small.sh.

DFlash retired from active service. DFlash (draft-dflash) was tested initially but did not deliver the desired speed gains under the frank-default-small workload (see ngram-mod benchmarks below). The Alittlehammmer/Qwen3.6-35B-A3B-DFlash draft GGUF remains in /home/dank/models/ as reference material only.

The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:

• FP8 KV cache is valid on CUDA 12.
• FP8 weights compute through a fallback path; no native FP8 tensor cores.
• The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.

Working DFlash flags for this server:

--spec-method dflash
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
--spec-tokens 8
--kv-cache-dtype fp8

Example DFlash + AWQ on the RTX 3090:

docker run --rm --name vllm-qwen36-dflash \
    --gpus all --ipc=host \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    docker.io/vllm/vllm-openai:latest \
    cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
    --attention-backend flash_attn \
    --gpu-memory-utilization 0.95

Deployment on Dankclaw (AMD Strix Halo gfx1151)

Dankclaw now runs llama.cpp as its runtime for frank-default-large (port 7770), serving the HauhauCS Qwen3.6-27B uncensored Q8_K_P model with a DFlash draft model (draft-dflash) on the ROCm (HIP) backend. The earlier vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime (native MTP) was retired because the model had problems.

Current facts:

• Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive Q8_K_P (dense 27B).
• Draft model: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp Q8_0 (block size 16).
• Spec command: --spec-type draft-dflash with --spec-draft-n-max 8.
• Backend --device ROCm0; --gpu-layers all --fit off; KV q8_0 K / f16 V.
• Context: 3 x 200700 = 602100; batch 2048 / ubatch 512.

Benchmark result: ROCm beats Vulkan, and DFlash-alone generates faster than a DFlash+ngram layer at 4k. See frank-default-large-benchmark for full numbers and the memory-bound batch/ubatch limits.

Notes

• Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
• DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
• The old --swap-space flag no longer exists in current vLLM. Use --kv-offloading-size.
• --speculative-config is the preferred CLI. Legacy --spec-method, --spec-model, and --spec-tokens flags still exist.
• Use rejection_sample_method default strict for lossless output.

See Also

• Quantization by Hardware Architecture
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw
• Dankclaw System Specs
• Dankserver System Specs
• VLLM Overview
• deployment