serverAppConfigs/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

---
type: reference
description: "DFlash and DSpark are the two newest speculative decoding methods."
tags:
  - llm
  - speculative
  - dflash
timestamp: 2026-09-03
---

# DFlash and DSpark Speculative Decoding

## Overview

DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.

DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.

## How DFlash Works

DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:

- The draft model reuses the embedding and LM head from the target model.
- Only the intermediate layers are trained.
- It fuses target-model hidden features into the KV projections of every draft layer.
- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.

Results from the DFlash paper (arXiv:2602.06036):

- Over 6x lossless speedup on Qwen3-8B.
- Up to 2.5x higher speedup than EAGLE-3.
- Diffusion drafting cost is flat, so block size does not increase cost.

## How DSpark Works

DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.

Results from the DSpark paper:

- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
- Useful when parallel drafts decay in acceptance rate.

Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.

## Runtime Support

| Runtime | DFlash | DSpark | MTP |
|---------|--------|--------|-----|
| vLLM | yes (v0.20.1+) | yes | yes |
| llama.cpp | yes | yes | yes |
| SGLang | yes | no | yes |

### vLLM

vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object.

DFlash on vLLM:

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
    --attention-backend flash_attn \
    --max-num-batched-tokens 32768
```

Legacy flags also work:

```
vllm serve Qwen/Qwen3.6-35B-A3B \
    --spec-method dflash \
    --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
    --spec-tokens 8
```

DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.

DSpark on vLLM:

```
vllm serve <target-model> \
    --speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'
```

### llama.cpp

llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.

`--spec-type` accepts a **comma-separated list** of speculative implementations. `common_speculative_init` (in `common/speculative.cpp`) builds a vector of implementations and chains drafting through all of them (`common_speculative_draft_params.drafting` is "used to chain the drafts through all the available implementations"). This means you can **layer ngram-mod on top of a DFlash draft model** in one server process:

```
llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <dflash-draft-GGUF> \
    --spec-type draft-dflash,ngram-mod
```

The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).

### Chaining laws (measured on this fleet, 2026-09 → 10)

`--spec-type` takes a comma list, and the gate is `ctx_dft` in
`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:

- A chain may hold ONE context-backed drafter plus any number of ngram types.
  `draft-dflash,ngram-mod` is legal and measured: 2.6x the drafter alone on
  prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64
  window.
- Two context-backed types cannot share one process: `draft-mtp,draft-dflash`
  dies at load with `context type MTP requested but model doesn't contain MTP
  layers` — MTP builds its context against whatever the draft context holds,
  so it met the DRAFTER's file, not a missing head. Pin that in a seam so
  nobody "fixes the flag" later.
- Internal order is fixed (ngram types first, then drafters); CLI order is
  irrelevant.
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
  on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
  and MTP verification depends on that rollback, so acceptance drops on ALL
  traffic even where the matcher fires. Measure chains per architecture,
  never by analogy.
- ngram-mod prints NO startup banner — the load log shows only the drafter's
  `adding speculative implementation` line. Never conclude "chain not
  engaged" from the load log; identify behaviourally (`draft_n` > 0 only
  where text repeats) or by the per-task `draft acceptance = ... mean len`
  line. It costs ~64 MiB of host-RAM hash table.

### Drafter KV: read the window policy from the header

The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)`
(`common/speculative.cpp`) — there is no flag to cap it — so a drafter that
declares NO sliding window caches the whole target window on whatever device
`--spec-draft-device` names. DFlash declares `sliding_window = 2048` (~40 MiB
of draft KV); DSpark declares none: 5 layers x 2 x 8 kv heads x 128 dim x 2 B
= 20 KiB per position, i.e. **5.0 GiB at 262144** — enough to kill the first
draft graph capture with `ROCm error: out of memory` AFTER the model loaded
and `/health` answered (measured 2026-09-16). Read `sliding_window` out of
the drafter's GGUF header before sizing any spec-decode deployment, and
prefer a skewed `-ts` over a context cut when the full window matters.

DFlash on llama.cpp:

```
llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dflash \
    --spec-draft-n-max 16
```

DSpark on llama.cpp:

```
llama-server --hf-repo <target-GGUF> \
    --spec-draft-hf <draft-GGUF> \
    --spec-type draft-dspark
```

MTP on llama.cpp:

```
llama-server --hf-repo <target-with-MTP> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2
```

Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.

### MTP drafter availability for Qwen3.6-27B

MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673).

Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:

| File | Architecture | Tensors | MTP tensors |
|------|--------------|---------|-------------|
| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 |
| `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 |

The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:

```
llama-server --model <HauhauCS-Q8_K_P.gguf> \
    --spec-draft-model <MTP-only-draft.gguf> \
    --spec-type draft-mtp \
    --spec-draft-n-max 2
```

Compatible MTP-only draft GGUFs on Hugging Face:

| Repo | Notes |
|------|-------|
| `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` |
| `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` |
| `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) |
| `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) |

Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.

On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).

## Draft Models

DFlash draft model sizes for the models on these servers:

| Target model | DFlash draft | Draft size |
|--------------|--------------|------------|
| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B |
| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B |
| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B |
| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B |

The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.

## Deployment on Dankserver (NVIDIA RTX 3090)

The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.

> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.

The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:

- FP8 KV cache is valid on CUDA 12.
- FP8 weights compute through a fallback path; no native FP8 tensor cores.
- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.

Working DFlash flags for this server:

```
--spec-method dflash
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
--spec-tokens 8
--kv-cache-dtype fp8
```

Example DFlash + AWQ on the RTX 3090:

```
docker run --rm --name vllm-qwen36-dflash \
    --gpus all --ipc=host \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    docker.io/vllm/vllm-openai:latest \
    cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
    --attention-backend flash_attn \
    --gpu-memory-utilization 0.95
```

## Deployment on Dankclaw (AMD Strix Halo gfx1151)

`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[frank-dragoon]].

History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.

## A/B protocol on a live llama.cpp endpoint

Verdicts are **traffic-shaped**: a proposer can only draft text it has seen,
so a single prompt shape answers the wrong question. Use fixed shapes:

| Shape | Prompt | Who it can favour |
|---|---|---|
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
| identity | short deterministic question | controls for lossless verification |

Design the echo passage so no long span repeats INSIDE it — otherwise you
measure the matcher against itself. For `structured`, give each line a unique
id: repeated lines make it a negative control, not a repetition test.

Protocol:

1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait
   for the port (a retired predecessor's published-port proxy leaves the
   endpoint port in TIME_WAIT — the bind fails with address-in-use long after
   the unit stopped).
2. Per config: write an env file with the overrides, start the serve script
   with `FRANK_*_ENV_FILE`, poll `/health`, measure. Ready-made harness:
   `scripts/bench-spec-ab.py <env-file> <serve-script>`.
3. **Restore the unit in a `finally`** and shout if it does not come back
   healthy. A crashed harness that leaves production down is worse than no
   harness.
4. **Prove losslessness**: one fixed prompt at temp 0 to every config, diff
   the greedy text. Speculation only buys speed; differing texts are a
   correctness bug. Empty answers (`content: null` when reasoning ate the
   budget) make the comparison vacuous — bench clients must fall back to
   `reasoning_content`, and echo/structured prompts should disable thinking
   per request: `chat_template_kwargs: {"enable_thinking": false}`.
5. **Sweep the depth, not just the mode** — draft depth is model-specific and
   single-valued; deeper often loses (measured: at n=7 a third of proposals
   survived, at n=3 more than half).
6. **Sweep the ngram window in the same run as the configs**: too short a
   `--spec-ngram-mod-n-min/-n-max` proposes on ordinary prose fragments —
   rejected, verification still paid (measured 43.9 → 36.0 tok/s).

Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
`predicted_per_second`; cumulative `/metrics`
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
while tokens flow = the proposer is NOT engaged, whatever the flags say. A
zero-draft config is a useful negative control: it proves the other config's
drafts came from the proposer under test.

Harness hygiene: absolute script paths (a relative path plus `cd` to its own
dir double-counts the prefix); `pkill -x`, never `-f` (the pattern matches
your own command line); nested f-strings with backslash escapes are a
SyntaxError on the fleet's Python (<3.12 semantics) — build log lines with
concatenation. Report `decode tok/s` per shape plus acceptance and mean
accepted length; a config that wins on echo and loses on prose is a traffic
decision, not a winner. Wire the winner into the deployed env file with the
measured table as its justification comment; record losing numbers with "do
not switch without re-measuring".

The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
slot split, quant rung — not just spec flags) is the transient systemd
drop-in procedure in [[frank-dragoon]] and [[frank-zealot]].

## Is it worth it? (arithmetic before deployment)

Answer against the INCUMBENT, not against zero.

**Step 0 — check whether the target already speculates.** Many engines ship a
drafter of their own; then your baseline is `A_incumbent / t_s`, not `1 / t_s`.
halogen-flash-server: MTP head ON by default — read its own accounting, e.g.
`serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
llama.cpp: `/metrics` counters and per-request `draft acceptance` lines.

**Step 1 — measure the two step times, never guess.** Target step `t_s` = 1 /
(client-visible tok/s ÷ commit-per-round); ask for the rate the operator
actually SEES (engine-internal decode-only numbers run ~1.7–2x optimistic).
Drafter step `d_s` from llama.cpp's per-task `eval time` lines when it is a
plain model on the same box.

**Step 2 — arithmetic:**

```
round_time = K * d_s + 1.1 * t_s        # K drafts proposed, K+1 verified in ONE target pass
A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
rate       = A / round_time             # compare to the incumbent
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
```

Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67
commit/round), `d_s = 12 ms` (27B IQ3_S drafter on the 3090) → break-even
`p ≈ 0.70` at K=3, `≈0.68` at K=5; a realistic cross-finetune `p = 0.4–0.6`
gives 17–20 tok/s, i.e. a LOSS. A drafter only ~3x faster per token cannot
pay for K sequential forwards.

**Step 3 — classify the drafter by per-round cost:**

| drafter | cost per round | needs |
|---|---|---|
| plain model (`-md`/`draft-simple`) | **K sequential forwards** | 10–50x faster than the target |
| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
| trained head (`draft-mtp`, EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |

Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.

**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
continuation agreement on identical prompts, two API calls, no weights moved.
llama.cpp can also synthesize the projection without any draft artifact:
`--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
usually sparse MoE, the worst case for drafting.

**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
engines refuse external drafters by design: halogen-flash-server's
`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
name is a deliberate 400 (a silent downgrade would make comparisons lie);
there is no device routing in its loopback token protocol, so a drafter can
never sit on another card. Pure llama.cpp DOES route the draft:
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
One process sees only the devices its backends were built with — CUDA+Vulkan
is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
coexist); a CUDA-only build has no `Vulkan0` to draft with.

**Step 6 — cross-model pairs need a byte-identical tokenizer.** Speculation
compares token IDs, so draft and target must map id → same string. Verify
cheaply from the GGUF headers (first ~8 MB over HTTP, no full download):
parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and sha256
over the first 2000 tokens —

```
vocab=248320 merges=247587 model=gpt2 pre=qwen35
sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable
```

Identity is necessary, NOT sufficient — the drafter still has to be cheap and
accepted. Report in this order: does the target already draft; break-even
`p` at K=1,3,7 versus the incumbent; the realistic band and resulting rate;
whether the drafter fits (VRAM) and the attachment surface exists; the one
measurement that would settle it.

## Notes

- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
- Use `rejection_sample_method` default `strict` for lossless output.

## See Also

- [[Quantization by Hardware Architecture]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle