--- type: reference description: "DFlash and DSpark are the two newest speculative decoding methods." tags: - llm - speculative - dflash timestamp: 2026-09-03 --- # DFlash and DSpark Speculative Decoding ## Overview DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads. DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run. ## How DFlash Works DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points: - The draft model reuses the embedding and LM head from the target model. - Only the intermediate layers are trained. - It fuses target-model hidden features into the KV projections of every draft layer. - A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens. Results from the DFlash paper (arXiv:2602.06036): - Over 6x lossless speedup on Qwen3-8B. - Up to 2.5x higher speedup than EAGLE-3. - Diffusion drafting cost is flat, so block size does not increase cost. ## How DSpark Works DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk. Results from the DSpark paper: - 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic. - Useful when parallel drafts decay in acceptance rate. Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler. ## Runtime Support | Runtime | DFlash | DSpark | MTP | |---------|--------|--------|-----| | vLLM | yes (v0.20.1+) | yes | yes | | llama.cpp | yes | yes | yes | | SGLang | yes | no | yes | ### vLLM vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object. DFlash on vLLM: ``` vllm serve Qwen/Qwen3.6-35B-A3B \ --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \ --attention-backend flash_attn \ --max-num-batched-tokens 32768 ``` Legacy flags also work: ``` vllm serve Qwen/Qwen3.6-35B-A3B \ --spec-method dflash \ --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \ --spec-tokens 8 ``` DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash. DSpark on vLLM: ``` vllm serve \ --speculative-config '{"method": "dspark", "model": "", "num_speculative_tokens": 15}' ``` ### llama.cpp llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP. `--spec-type` accepts a **comma-separated list** of speculative implementations. `common_speculative_init` (in `common/speculative.cpp`) builds a vector of implementations and chains drafting through all of them (`common_speculative_draft_params.drafting` is "used to chain the drafts through all the available implementations"). This means you can **layer ngram-mod on top of a DFlash draft model** in one server process: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dflash,ngram-mod ``` The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+). ### Chaining laws (measured on this fleet, 2026-09 → 10) `--spec-type` takes a comma list, and the gate is `ctx_dft` in `common/speculative.cpp` (`common_speculative_init_result`): **draftless** types (the ngram family — simple, map-k, map-k4v, mod, cache) are added unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences: - A chain may hold ONE context-backed drafter plus any number of ngram types. `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64 window. - Two context-backed types cannot share one process: `draft-mtp,draft-dflash` dies at load with `context type MTP requested but model doesn't contain MTP layers` — MTP builds its context against whatever the draft context holds, so it met the DRAFTER's file, not a missing head. Pin that in a seam so nobody "fixes the flag" later. - Internal order is fixed (ngram types first, then drafters); CLI order is irrelevant. - **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod` on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 % block-copy (2026-10-03): adding the ngram layer logs `recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support`, and MTP verification depends on that rollback, so acceptance drops on ALL traffic even where the matcher fires. Measure chains per architecture, never by analogy. - ngram-mod prints NO startup banner — the load log shows only the drafter's `adding speculative implementation` line. Never conclude "chain not engaged" from the load log; identify behaviourally (`draft_n` > 0 only where text repeats) or by the per-task `draft acceptance = ... mean len` line. It costs ~64 MiB of host-RAM hash table. ### Drafter KV: read the window policy from the header The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is no flag to cap it — so a drafter that declares NO sliding window caches the whole target window on whatever device `--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first draft graph capture with `ROCm error: out of memory` AFTER the model loaded and `/health` answered (measured 2026-09-16). Read `sliding_window` out of the drafter's GGUF header before sizing any spec-decode deployment, and prefer a skewed `-ts` over a context cut when the full window matters. DFlash on llama.cpp: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dflash \ --spec-draft-n-max 16 ``` DSpark on llama.cpp: ``` llama-server --hf-repo \ --spec-draft-hf \ --spec-type draft-dspark ``` MTP on llama.cpp: ``` llama-server --hf-repo \ --spec-type draft-mtp \ --spec-draft-n-max 2 ``` Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding. ### MTP drafter availability for Qwen3.6-27B MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673). Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse: | File | Architecture | Tensors | MTP tensors | |------|--------------|---------|-------------| | `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 | | `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 | The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar: ``` llama-server --model \ --spec-draft-model \ --spec-type draft-mtp \ --spec-draft-n-max 2 ``` Compatible MTP-only draft GGUFs on Hugging Face: | Repo | Notes | |------|-------| | `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` | | `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` | | `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) | | `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) | Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls. On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]). ## Draft Models DFlash draft model sizes for the models on these servers: | Target model | DFlash draft | Draft size | |--------------|--------------|------------| | Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B | | Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B | | Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B | | Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B | The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput. ## Deployment on Dankserver (NVIDIA RTX 3090) The Dankserver llama.cpp deployment (`frank-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware. > **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: - FP8 KV cache is valid on CUDA 12. - FP8 weights compute through a fallback path; no native FP8 tensor cores. - The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target. Working DFlash flags for this server: ``` --spec-method dflash --spec-model z-lab/Qwen3.6-35B-A3B-DFlash --spec-tokens 8 --kv-cache-dtype fp8 ``` Example DFlash + AWQ on the RTX 3090: ``` docker run --rm --name vllm-qwen36-dflash \ --gpus all --ipc=host \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ docker.io/vllm/vllm-openai:latest \ cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \ --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \ --attention-backend flash_attn \ --gpu-memory-utilization 0.95 ``` ## Deployment on Dankclaw (AMD Strix Halo gfx1151) `frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]]. History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. ## A/B protocol on a live llama.cpp endpoint Verdicts are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question. Use fixed shapes: | Shape | Prompt | Who it can favour | |---|---|---| | prose | 1k filler + "reply with one sentence" | model-backed drafters only | | echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) | | structured | repetitive output, no prompt overlap | ngram, if spans really repeat | | identity | short deterministic question | controls for lossless verification | Design the echo passage so no long span repeats INSIDE it — otherwise you measure the matcher against itself. For `structured`, give each line a unique id: repeated lines make it a negative control, not a repetition test. Protocol: 1. Stop the unit (`sudo systemctl stop `), `pkill -x llama-server`; wait for the port (a retired predecessor's published-port proxy leaves the endpoint port in TIME_WAIT — the bind fails with address-in-use long after the unit stopped). 2. Per config: write an env file with the overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness: `scripts/bench-spec-ab.py `. 3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness. 4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff the greedy text. Speculation only buys speed; differing texts are a correctness bug. Empty answers (`content: null` when reasoning ate the budget) make the comparison vacuous — bench clients must fall back to `reasoning_content`, and echo/structured prompts should disable thinking per request: `chat_template_kwargs: {"enable_thinking": false}`. 5. **Sweep the depth, not just the mode** — draft depth is model-specific and single-valued; deeper often loses (measured: at n=7 a third of proposals survived, at n=3 more than half). 6. **Sweep the ngram window in the same run as the configs**: too short a `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments — rejected, verification still paid (measured 43.9 → 36.0 tok/s). Read the right counters — per request `timings.draft_n` / `draft_n_accepted` / `predicted_per_second`; cumulative `/metrics` `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`; per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0` while tokens flow = the proposer is NOT engaged, whatever the flags say. A zero-draft config is a useful negative control: it proves the other config's drafts came from the proposer under test. Harness hygiene: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches your own command line); nested f-strings with backslash escapes are a SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with concatenation. Report `decode tok/s` per shape plus acceptance and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Wire the winner into the deployed env file with the measured table as its justification comment; record losing numbers with "do not switch without re-measuring". The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type, slot split, quant rung — not just spec flags) is the transient systemd drop-in procedure in [[frank-large-benchmark]] and [[frank-small-benchmark]]. ## Is it worth it? (arithmetic before deployment) Answer against the INCUMBENT, not against zero. **Step 0 — check whether the target already speculates.** Many engines ship a drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`. halogen-flash-server: MTP head ON by default — read its own accounting, e.g. `serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`. llama.cpp: `/metrics` counters and per-request `draft acceptance` lines. **Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 / (client-visible tok/s ÷ commit-per-round); ask for the rate the operator actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic). Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a plain model on the same box. **Step 2 — arithmetic:** ``` round_time = K * d_s + 1.1 * t_s # K drafts proposed, K+1 verified in ONE target pass A = (1 - p^(K+1)) / (1 - p) # expected accepted tokens, p = per-position acceptance rate = A / round_time # compare to the incumbent break-even : A > rate_incumbent * (K*d_s + 1.1*t_s) ``` Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67 commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even `p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6` gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot pay for K sequential forwards. **Step 3 — classify the drafter by per-round cost:** | drafter | cost per round | needs | |---|---|---| | plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target | | n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) | | trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT | Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk. **Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2 at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5 tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy for `p` at temp 0 when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved. llama.cpp can also synthesize the projection without any draft artifact: `--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an 18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are usually sparse MoE, the worst case for drafting. **Step 5 — check the attachment surface BEFORE the arithmetic.** Closed engines refuse external drafters by design: halogen-flash-server's `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie); there is no device routing in its loopback token protocol, so a drafter can never sit on another card. Pure llama.cpp DOES route the draft: `--spec-draft-device/-devd` (default follows `--device`), `-ngld`, `-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. One process sees only the devices its backends were built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no `Vulkan0` to draft with. **Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation compares token IDs, so draft and target must map id → same string. Verify cheaply from the GGUF headers (first ~8 MB over HTTP, no full download): parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256 over the first 2000 tokens — ``` vocab=248320 merges=247587 model=gpt2 pre=qwen35 sha256(first 2000 tokens)=8619c68fe4466d3a # identical => ids are interchangeable ``` Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted. Report in this order: does the target already draft; break-even `p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate; whether the drafter fits (VRAM) and the attachment surface exists; the one measurement that would settle it. ## Notes - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS. - DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. - The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`. - `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist. - Use `rejection_sample_method` default `strict` for lossless output. ## See Also - [[Quantization by Hardware Architecture]] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] - [[Running Qwen3.6 27B FP8 on Dankclaw]] - [[Dankclaw System Specs]] - [[Dankserver System Specs]] - [[VLLM Overview]] - [[deployment]] - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic - [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands - [[frank-large-benchmark]] and [[frank-small-benchmark]] — the single-GPU env drop-in A/B vehicle