# DFlash and DSpark Speculative Decoding ## Overview DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads. DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run. ## How DFlash Works DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points: - The draft model reuses the embedding and LM head from the target model. - Only the intermediate layers are trained. - It fuses target-model hidden features into the KV projections of every draft layer. - A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens. Results from the DFlash paper (arXiv:2602.06036): - Over 6x lossless speedup on Qwen3-8B. - Up to 2.5x higher speedup than EAGLE-3. - Diffusion drafting cost is flat, so block size does not increase cost. ## How DSpark Works DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk. Results from the DSpark paper: - 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic. - Useful when parallel drafts decay in acceptance rate. Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler. ## Runtime Support | Runtime | DFlash | DSpark | MTP | |---------|--------|--------|-----| | vLLM | yes (v0.20.1+) | yes | yes | | llama.cpp | yes | yes | yes | | SGLang | yes | no | yes | ### vLLM vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object. DFlash on vLLM: ``` vllm serve Qwen/Qwen3.6-35B-A3B \ --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \ --attention-backend flash_attn \ --max-num-batched-tokens 32768 ``` Legacy flags also work: ``` vllm serve Qwen/Qwen3.6-35B-A3B \ --spec-method dflash \ --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \ --spec-tokens 8 ``` DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash. DSpark on vLLM: ``` vllm serve \ --speculative-config '{"method": "dspark", "model": "", "num_speculative_tokens": 15}' ``` ### llama.cpp llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP. `--spec-type` accepts a **comma-separated list** of speculative implementations. `common_speculative_init` (in `common/speculative.cpp`) builds a vector of implementations and chains drafting through all of them (`common_speculative_draft_params.drafting` is "used to chain the drafts through all the available implementations"). This means you can **layer ngram-mod on top of a DFlash draft model** in one server process: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dflash,ngram-mod ``` The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+). DFlash on llama.cpp: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dflash \ --spec-draft-n-max 16 ``` DSpark on llama.cpp: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dspark ``` MTP on llama.cpp: ``` llama-server --hf-repo \ --spec-type draft-mtp \ --spec-draft-n-max 2 ``` Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding. ### MTP drafter availability for Qwen3.6-27B MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673). Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse: | File | Architecture | Tensors | MTP tensors | |------|--------------|---------|-------------| | `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 | | `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 | The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar: ``` llama-server --model \ --spec-draft-model \ --spec-type draft-mtp \ --spec-draft-n-max 2 ``` Compatible MTP-only draft GGUFs on Hugging Face: | Repo | Notes | |------|-------| | `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` | | `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` | | `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) | | `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) | Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls. On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]). ## Draft Models DFlash draft model sizes for the models on these servers: | Target model | DFlash draft | Draft size | |--------------|--------------|------------| | Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B | | Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B | | Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B | | Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B | The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput. ## Deployment on Dankserver (NVIDIA RTX 3090) ## Deployment on Dankserver (NVIDIA RTX 3090) The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`. > **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: - FP8 KV cache is valid on CUDA 12. - FP8 weights compute through a fallback path; no native FP8 tensor cores. - The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target. Working DFlash flags for this server: ``` --spec-method dflash --spec-model z-lab/Qwen3.6-35B-A3B-DFlash --spec-tokens 8 --kv-cache-dtype fp8 ``` Example DFlash + AWQ on the RTX 3090: ``` docker run --rm --name vllm-qwen36-dflash \ --gpus all --ipc=host \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ docker.io/vllm/vllm-openai:latest \ cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \ --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \ --attention-backend flash_attn \ --gpu-memory-utilization 0.95 ``` ## Deployment on Dankclaw (AMD Strix Halo gfx1151) Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14. Current facts: - Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B). - Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar). - Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`. - Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V. - Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192. Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only. ## Notes - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS. - DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. - The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`. - `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist. - Use `rejection_sample_method` default `strict` for lossless output. ## See Also - [[Quantization by Hardware Architecture]] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] - [[Running Qwen3.6 27B FP8 on Dankclaw]] - [[Dankclaw System Specs]] - [[Dankserver System Specs]] - [[VLLM Overview]] - [[deployment]]