docs: add DFlash/DSpark speculative decoding research + quantization-by-hardware guide

- DFlash DSpark Speculative Decoding.md: verified method details (DFlash block diffusion from Z Lab arXiv:2602.06036; DSpark semi-autoregressive from DeepSeek arXiv:2607.05147), runtime support matrix (vLLM/llama.cpp/SGLang), exact vLLM --speculative-config recipes (incl. dflash/dspark), llama.cpp --spec-type recipes with sidecar auto-discovery, draft model table (z-lab Qwen3.6-27B-DFlash 2B, Qwen3.6-35B-A3B-DFlash 0.4B), deployment guidance for RTX 3090 (AWQ+Dflash) and Strix Halo (ROCmFPX MTP), measured 45-50 t/s MTP decode on Strix Halo - Quantization by Hardware Architecture.md: FP8/INT8/INT4/GPTQ/AWQ/GGUF/FP4/ ROCmFP4 support per NVIDIA (Turing/Ampere/Ada/Hopper/Blackwell), AMD (CDNA MI250/MI300/MI350, RDNA 3/3.5/4), and CPU; recommended picks per Dankserver (AWQ/GPTQ 4-bit) and Dankclaw (ROCmFP4/Q4_K_M) - index.md: link both new pages under Speculative Decoding & Quantization
2026-08-10 21:09:17 -07:00
1 parent 90c4883f89
commit 993b6af576
@@ -0,0 +1,181 @@
+# DFlash and DSpark Speculative Decoding
+
+## Overview
+
+DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.
+
+DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.
+
+## How DFlash Works
+
+DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:
+
+- The draft model reuses the embedding and LM head from the target model.
+- Only the intermediate layers are trained.
+- It fuses target-model hidden features into the KV projections of every draft layer.
+- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.
+
+Results from the DFlash paper (arXiv:2602.06036):
+
+- Over 6x lossless speedup on Qwen3-8B.
+- Up to 2.5x higher speedup than EAGLE-3.
+- Diffusion drafting cost is flat, so block size does not increase cost.
+
+## How DSpark Works
+
+DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.
+
+Results from the DSpark paper:
+
+- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
+- Useful when parallel drafts decay in acceptance rate.
+
+Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.
+
+## Runtime Support
+
+| Runtime | DFlash | DSpark | MTP |
+|---------|--------|--------|-----|
+| vLLM | yes (v0.20.1+) | yes | yes |
+| llama.cpp | yes | yes | yes |
+| SGLang | yes | no | yes |
+
+### vLLM
+
+vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object.
+
+DFlash on vLLM:
+
+```
+vllm serve Qwen/Qwen3.6-35B-A3B \
+    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
+    --attention-backend flash_attn \
+    --max-num-batched-tokens 32768
+```
+
+Legacy flags also work:
+
+```
+vllm serve Qwen/Qwen3.6-35B-A3B \
+    --spec-method dflash \
+    --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
+    --spec-tokens 8
+```
+
+DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.
+
+DSpark on vLLM:
+
+```
+vllm serve <target-model> \
+    --speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'
+```
+
+### llama.cpp
+
+llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.
+
+DFlash on llama.cpp:
+
+```
+llama-server --hf-repo <target-GGUF> \
+    --spec-draft-hf <draft-GGUF> \
+    --spec-type draft-dflash \
+    --spec-draft-n-max 16
+```
+
+DSpark on llama.cpp:
+
+```
+llama-server --hf-repo <target-GGUF> \
+    --spec-draft-hf <draft-GGUF> \
+    --spec-type draft-dspark
+```
+
+MTP on llama.cpp:
+
+```
+llama-server --hf-repo <target-with-MTP> \
+    --spec-type draft-mtp \
+    --spec-draft-n-max 2
+```
+
+## Draft Models
+
+DFlash draft model sizes for the models on these servers:
+
+| Target model | DFlash draft | Draft size |
+|--------------|--------------|------------|
+| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B |
+| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B |
+| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B |
+| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B |
+
+The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.
+
+## Deployment on Dankserver (NVIDIA RTX 3090)
+
+Dankserver runs vLLM in a Podman container with the RTX 3090. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
+
+- FP8 KV cache is valid on CUDA 12.
+- FP8 weights compute through a fallback path; no native FP8 tensor cores.
+- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.
+
+Working DFlash flags for this server:
+
+```
+--spec-method dflash
+--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
+--spec-tokens 8
+--kv-cache-dtype fp8
+```
+
+Example DFlash + AWQ on the RTX 3090:
+
+```
+docker run --rm --name vllm-qwen36-dflash \
+    --gpus all --ipc=host \
+    -p 8000:8000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    docker.io/vllm/vllm-openai:latest \
+    cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
+    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
+    --attention-backend flash_attn \
+    --gpu-memory-utilization 0.95
+```
+
+## Deployment on Dankclaw (AMD Strix Halo gfx1151)
+
+Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results:
+
+- 45-50 t/s decode with MTP speculation.
+- ~37 t/s plain decode.
+- ~95% draft acceptance with `--spec-draft-n-max 2`.
+
+ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp.
+
+Ling-3.0-flash-related facts:
+
+- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active.
+- The MTP (NextN) head is preserved in the GGUF conversion.
+- The raulvidis GGUF repo targets Strix Halo (gfx1151).
+
+If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151.
+
+## Notes
+
+- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
+- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
+- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
+- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
+- Use `rejection_sample_method` default `strict` for lossless output.
+
+## See Also
+
+- [[Quantization by Hardware Architecture]]
+- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
+- [[Running Qwen3.6 27B FP8 on Dankclaw]]
+- [[Dankclaw System Specs]]
+- [[Dankserver System Specs]]
+- [[VLLM Overview]]
+- [[deployment]]