serverAppConfigs/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	DFlash and DSpark are the two newest speculative decoding methods.	llm speculative dflash	2026-09-03

DFlash and DSpark Speculative Decoding

Overview

DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.

DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both publish draft models for the Qwen3.5/Qwen3.6 families (table below); what any of them pairs with on this fleet is an era on frank-dragoon or frank-zealot.

How DFlash Works

DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:

• The draft model reuses the embedding and LM head from the target model.
• Only the intermediate layers are trained.
• It fuses target-model hidden features into the KV projections of every draft layer.
• A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.

Results from the DFlash paper (arXiv:2602.06036):

• Over 6x lossless speedup on Qwen3-8B.
• Up to 2.5x higher speedup than EAGLE-3.
• Diffusion drafting cost is flat, so block size does not increase cost.

How DSpark Works

DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.

Results from the DSpark paper:

• 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
• Useful when parallel drafts decay in acceptance rate.

Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.

Runtime Support

Runtime	DFlash	DSpark	MTP
vLLM	yes (v0.20.1+)	yes	yes
llama.cpp	yes	yes	yes
SGLang	yes	no	yes

vLLM

vLLM v0.20.1+ has core DFlash support. Use the --speculative-config JSON object.

DFlash on vLLM:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
    --attention-backend flash_attn \
    --max-num-batched-tokens 32768

Legacy flags also work:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --spec-method dflash \
    --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
    --spec-tokens 8

DFlash requires a non-causal attention backend. Set "attention_backend": "flash_attn" in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.

DSpark on vLLM:

vllm serve <target-model> \
    --speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'

llama.cpp

llama.cpp supports DFlash, DSpark, and MTP through --spec-type. When you pass --spec-draft-hf, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.

--spec-type accepts a comma-separated list of speculative implementations. common_speculative_init (in common/speculative.cpp) builds a vector of implementations and chains drafting through all of them (common_speculative_draft_params.drafting is "used to chain the drafts through all the available implementations"). This means you can layer ngram-mod on top of a DFlash draft model in one server process:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <dflash-draft-GGUF> \
    --spec-type draft-dflash,ngram-mod

The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream common/speculative.cpp (b10133+).

Chaining laws (measured on this fleet, 2026-09 → 10)

--spec-type takes a comma list, and the gate is ctx_dft in common/speculative.cpp (common_speculative_init_result): draftless types (the ngram family — simple, map-k, map-k4v, mod, cache) are added unconditionally, while context-backed types (draft-mtp, draft-dflash, draft-dspark, draft-eagle3) all require a draft CONTEXT. Consequences:

• A chain may hold ONE context-backed drafter plus any number of ngram types. draft-dflash,ngram-mod is legal and measured: 2.6x the drafter alone on prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64 window.
• Two context-backed types cannot share one process: draft-mtp,draft-dflash dies at load with context type MTP requested but model doesn't contain MTP layers — MTP builds its context against whatever the draft context holds, so it met the DRAFTER's file, not a missing head. Pin that in a seam so nobody "fixes the flag" later.
• Internal order is fixed (ngram types first, then drafters); CLI order is irrelevant.
• A legal chain can still lose on a recurrent model. draft-mtp,ngram-mod on the qwen35 hybrid loses on both traffic shapes (2026-10-03): adding the ngram layer logs recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support, and MTP verification depends on that rollback, so acceptance drops on ALL traffic even where the matcher fires. The measured size of each loss is frank-dragoon's (Laws). Measure chains per architecture, never by analogy.
• ngram-mod prints NO startup banner — the load log shows only the drafter's adding speculative implementation line. Never conclude "chain not engaged" from the load log; identify behaviourally (draft_n > 0 only where text repeats) or by the per-task draft acceptance = ... mean len line. It costs ~64 MiB of host-RAM hash table.

Drafter KV: read the window policy from the header

The draft context is built with cparams.n_ctx = llama_n_ctx(ctx_tgt) (common/speculative.cpp) — there is no flag to cap it — so a drafter that declares NO sliding window caches the whole target window on whatever device --spec-draft-device names. DFlash declares sliding_window = 2048 (~40 MiB of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position, i.e. 5.0 GiB at 262144 — enough to kill the first draft graph capture with ROCm error: out of memory AFTER the model loaded and /health answered (measured 2026-09-16). Read sliding_window out of the drafter's GGUF header before sizing any spec-decode deployment, and prefer a skewed -ts over a context cut when the full window matters.

DFlash on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dflash \
    --spec-draft-n-max 16

DSpark on llama.cpp:

llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dspark

MTP on llama.cpp:

llama-server --hf-repo <target-with-MTP> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2

Note: the flag was renamed from --spec-type mtp to --spec-type draft-mtp in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.

MTP drafter availability for Qwen3.6-27B

MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's draft-mtp implementation is fully wired for the qwen35 and qwen35moe architectures (PR #22673).

Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:

File	Architecture	Tensors	MTP tensors
Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf	qwen35	851	0
Qwen3.6-27B-DFlash-Q8_0.gguf (draft)	dflash	58	0

The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:

llama-server --model <HauhauCS-Q8_K_P.gguf> \
    --spec-draft-model <MTP-only-draft.gguf> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2

Compatible MTP-only draft GGUFs on Hugging Face:

Repo	Notes
a4lg/Qwen3.6-27B-MTP-ONLY-GGUF	MTP-only subset of the base model; pairs with derivative/abliterated targets via --spec-draft-model
lym00/Qwen3.6-27B-MTP-ONLY-GGUF	Same purpose; built from convert_hf_to_gguf.py --mtp
ggml-org/Qwen3.6-27B-MTP-GGUF	Full model with MTP head (replaces the target, not a sidecar)
hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF	HauhauCS Q4_K_P with the MTP head grafted in (replaces the target)

Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.

On the mid-August 2026 Dankclaw shape draft-dflash was the benchmarked winner at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those three figures are that retired llama.cpp era's property (frank-dragoon owns the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see deployment).

Draft Models

DFlash draft model sizes for the models on these servers:

Target model	DFlash draft	Draft size
Qwen3.6-35B-A3B	z-lab/Qwen3.6-35B-A3B-DFlash	0.4B
Qwen3.6-27B	z-lab/Qwen3.6-27B-DFlash	2B
Qwen3.5-35B-A3B	z-lab/Qwen3.5-35B-A3B-DFlash	0.4B
Qwen3.5-27B	z-lab/Qwen3.5-27B-DFlash	2B

The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.

DFlash on an RTX 3090 (Ampere) — the archived vLLM path

DFlash was tried on the 3090 line, lost there to the ngram chain, and was never kept deployed; that card's payloads, shapes and rates are frank-zealot's (its Eras rows are the llama.cpp arms the draft was tested against). What transfers is the Ampere arithmetic — compute capability 8.6, so:

• FP8 KV cache is valid on CUDA 12.
• FP8 weights compute through a fallback path; no native FP8 tensor cores.
• The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.

Working DFlash flags for this server:

--spec-method dflash
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
--spec-tokens 8
--kv-cache-dtype fp8

Example DFlash + AWQ on the RTX 3090:

docker run --rm --name vllm-qwen36-dflash \
    --gpus all --ipc=host \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    docker.io/vllm/vllm-openai:latest \
    cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
    --attention-backend flash_attn \
    --gpu-memory-utilization 0.95

DFlash and DSpark on Strix Halo (gfx1151) — what killed them

Both context-backed drafters died the same way on the iGPU: under parallel load. DSpark took ROCm error: an illegal memory access inside the shared draft context with two concurrent 16k prompts; DFlash corrupted multi-slot into premature EOS and was retired first. The timeline, the crash class, the "concurrency is the trigger, not size" finding and the PARALLEL=1 workaround are on qwen3.8-spec-matrix; the shape they were measured on is frank-dragoon's mid-August 2026 era.

A/B protocol on a live llama.cpp endpoint

Verdicts are traffic-shaped: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question. Use fixed shapes:

Shape	Prompt	Who it can favour
prose	1k filler + "reply with one sentence"	model-backed drafters only
echo	~1.5k passage of UNIQUE lines + "repeat verbatim"	ngram (output IS the prompt)
structured	repetitive output, no prompt overlap	ngram, if spans really repeat
identity	short deterministic question	controls for lossless verification

Design the echo passage so no long span repeats INSIDE it — otherwise you measure the matcher against itself. For structured, give each line a unique id: repeated lines make it a negative control, not a repetition test.

Protocol:

1. Stop the unit (sudo systemctl stop <svc>), pkill -x llama-server; wait for the port (a retired predecessor's published-port proxy leaves the endpoint port in TIME_WAIT — the bind fails with address-in-use long after the unit stopped).
2. Per config: write an env file with the overrides, start the serve script with FRANK_*_ENV_FILE, poll /health, measure. Ready-made harness: scripts/bench-spec-ab.py <env-file> <serve-script>.
3. Restore the unit in a finally and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
4. Prove losslessness: one fixed prompt at temp 0 to every config, diff the greedy text. Speculation only buys speed; differing texts are a correctness bug. Empty answers (content: null when reasoning ate the budget) make the comparison vacuous — bench clients must fall back to reasoning_content, and echo/structured prompts should disable thinking per request: chat_template_kwargs: {"enable_thinking": false}.
5. Sweep the depth, not just the mode — draft depth is model-specific and single-valued; deeper often loses (measured: at n=7 a third of proposals survived, at n=3 more than half).
6. Sweep the ngram window in the same run as the configs: too short a --spec-ngram-mod-n-min/-n-max proposes on ordinary prose fragments — rejected, verification still paid (measured 43.9 → 36.0 tok/s).

Read the right counters — per request timings.draft_n / draft_n_accepted / predicted_per_second; cumulative /metrics llamacpp:spec_decode_num_drafts_total / ..._accepted_tokens_total; per-task draft acceptance = ... (N accepted / M generated). draft_n == 0 while tokens flow = the proposer is NOT engaged, whatever the flags say. A zero-draft config is a useful negative control: it proves the other config's drafts came from the proposer under test.

Harness hygiene: absolute script paths (a relative path plus cd to its own dir double-counts the prefix); pkill -x, never -f (the pattern matches your own command line); nested f-strings with backslash escapes are a SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with concatenation. Report decode tok/s per shape plus acceptance and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Wire the winner into the deployed env file with the measured table as its justification comment; record losing numbers with "do not switch without re-measuring".

The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type, slot split, quant rung — not just spec flags) is the transient systemd drop-in procedure in frank-dragoon and frank-zealot.

Is it worth it? (arithmetic before deployment)

Answer against the INCUMBENT, not against zero.

Step 0 — check whether the target already speculates. Many engines ship a drafter of their own; then your baseline is A_incumbent / t_s, not 1 / t_s. halogen-flash-server: MTP head ON by default — read its own accounting, e.g. serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round. llama.cpp: /metrics counters and per-request draft acceptance lines.

Step 1 — measure the two step times, never guess. Target step t_s = 1 / (client-visible tok/s ÷ commit-per-round); ask for the rate the operator actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic). Drafter step d_s from llama.cpp's per-task eval time lines when it is a plain model on the same box.

Step 2 — arithmetic:

round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
rate       = A / round_time             # compare to the incumbent
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)

Worked example that decided a real case: t_s = 67 ms (25 tok/s, 1.67 commit/round), d_s = 12 ms (27B IQ3_S drafter on the 3090) → break-even p ≈ 0.70 at K=3, ≈0.68 at K=5; a realistic cross-finetune p = 0.4–0.6 gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot pay for K sequential forwards.

Step 3 — classify the drafter by per-round cost:

drafter	cost per round	needs
plain model (-md/draft-simple)	K sequential forwards	10–50x faster than the target
n-gram / prompt-lookup (ngram-*)	~0, no model	repeated context (agent edit loops)
trained head (draft-mtp, EAGLE/DFlash)	~1 pass	the head must exist FOR THAT CHECKPOINT

Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.

Step 4 — acceptance reality checks. Treat 40–60 % as the honest band for a matched pair; the arm that anchors the bottom of it on this fleet is the depth-7 draft-dflash row of frank-medium-llamacpp-dflash2's spec table (accepted share, rate and VRAM for every depth are in that table). Cheapest proxy for p at temp 0 when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved. llama.cpp can also synthesize the projection without any draft artifact: --spec-synth-rates P0,P1,... / --spec-synth-len L. Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS — that band's midpoint was worth 2.3× on the slow two-card layer-split target and a LOSS on the 25 tok/s target of the worked example above; fast targets are usually sparse MoE, the worst case for drafting.

Step 5 — check the attachment surface BEFORE the arithmetic. Closed engines refuse external drafters by design: halogen-flash-server's DRAFTER_CAP gates dflash2 on the loaded checkpoint and an unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie); there is no device routing in its loopback token protocol, so a drafter can never sit on another card. Pure llama.cpp DOES route the draft: --spec-draft-device/-devd (default follows --device), -ngld, -otd, -ctkd/-ctvd, --spec-draft-n-max (default 3), --spec-draft-p-min. One process sees only the devices its backends were built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no Vulkan0 to draft with.

Step 6 — cross-model pairs need a byte-identical tokenizer. Speculation compares token IDs, so draft and target must map id → same string. Verify cheaply from the GGUF headers (first ~8 MB over HTTP, no full download): parse tokenizer.ggml.tokens, print vocab size, merges, pre, and sha256 over the first 2000 tokens —

vocab=248320 merges=247587 model=gpt2 pre=qwen35
sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable

Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted. Report in this order: does the target already draft; break-even p at K=1,3,7 versus the incumbent; the realistic band and resulting rate; whether the drafter fits (VRAM) and the attachment surface exists; the one measurement that would settle it.

Notes

• Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
• DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
• The old --swap-space flag is retired; the live CPU-KV-offload knob pair is owned by RAM CPU Offloading in VLLM.
• --speculative-config is the preferred CLI. Legacy --spec-method, --spec-model, and --spec-tokens flags still exist.
• Use rejection_sample_method default strict for lossless output.

See Also

• Quantization by Hardware Architecture
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw
• Dankclaw System Specs
• Dankserver System Specs
• VLLM Overview
• deployment
• GGUF Compatibility — header parsing and custom tensor-type lock-in
• Llama Cpp Serving Sizing — slot-pool and drafter KV sizing arithmetic
• Llama Cpp Multi-Device Splits — where the drafter's KV actually lands
• frank-dragoon and frank-zealot — the single-GPU env drop-in A/B vehicle