Merge vllmScripts under llm/ (history-preserving import)
The vllmScripts repository (LLM inference fleet: serve scripts, systemd units, env tunables, deployment runbook, research docs) is merged into this repository. Its full history is grafted under llm/ with a two-parent merge commit so no commit is lost. A follow-up commit reorganizes llm/ into the repo layout: llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
This commit is contained in:
Frank committed
commit
918fe4eaec
82 files changed
+10855
No files matched your search
@@ -0,0 +1,218 @@
|
||||
# DFlash and DSpark Speculative Decoding
|
||||
|
||||
## Overview
|
||||
|
||||
DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.
|
||||
|
||||
DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.
|
||||
|
||||
## How DFlash Works
|
||||
|
||||
DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:
|
||||
|
||||
- The draft model reuses the embedding and LM head from the target model.
|
||||
- Only the intermediate layers are trained.
|
||||
- It fuses target-model hidden features into the KV projections of every draft layer.
|
||||
- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.
|
||||
|
||||
Results from the DFlash paper (arXiv:2602.06036):
|
||||
|
||||
- Over 6x lossless speedup on Qwen3-8B.
|
||||
- Up to 2.5x higher speedup than EAGLE-3.
|
||||
- Diffusion drafting cost is flat, so block size does not increase cost.
|
||||
|
||||
## How DSpark Works
|
||||
|
||||
DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.
|
||||
|
||||
Results from the DSpark paper:
|
||||
|
||||
- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
|
||||
- Useful when parallel drafts decay in acceptance rate.
|
||||
|
||||
Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.
|
||||
|
||||
## Runtime Support
|
||||
|
||||
| Runtime | DFlash | DSpark | MTP |
|
||||
|---------|--------|--------|-----|
|
||||
| vLLM | yes (v0.20.1+) | yes | yes |
|
||||
| llama.cpp | yes | yes | yes |
|
||||
| SGLang | yes | no | yes |
|
||||
|
||||
### vLLM
|
||||
|
||||
vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object.
|
||||
|
||||
DFlash on vLLM:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
|
||||
--attention-backend flash_attn \
|
||||
--max-num-batched-tokens 32768
|
||||
```
|
||||
|
||||
Legacy flags also work:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8
|
||||
```
|
||||
|
||||
DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.
|
||||
|
||||
DSpark on vLLM:
|
||||
|
||||
```
|
||||
vllm serve <target-model> \
|
||||
--speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'
|
||||
```
|
||||
|
||||
### llama.cpp
|
||||
|
||||
llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.
|
||||
|
||||
`--spec-type` accepts a **comma-separated list** of speculative implementations. `common_speculative_init` (in `common/speculative.cpp`) builds a vector of implementations and chains drafting through all of them (`common_speculative_draft_params.drafting` is "used to chain the drafts through all the available implementations"). This means you can **layer ngram-mod on top of a DFlash draft model** in one server process:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <dflash-draft-GGUF> \
|
||||
--spec-type draft-dflash,ngram-mod
|
||||
```
|
||||
|
||||
The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).
|
||||
|
||||
DFlash on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <draft-GGUF> \
|
||||
--spec-type draft-dflash \
|
||||
--spec-draft-n-max 16
|
||||
```
|
||||
|
||||
DSpark on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <draft-GGUF> \
|
||||
--spec-type draft-dspark
|
||||
```
|
||||
|
||||
MTP on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-with-MTP> \
|
||||
--spec-type draft-mtp \
|
||||
--spec-draft-n-max 2
|
||||
```
|
||||
|
||||
Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.
|
||||
|
||||
### MTP drafter availability for Qwen3.6-27B
|
||||
|
||||
MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673).
|
||||
|
||||
Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:
|
||||
|
||||
| File | Architecture | Tensors | MTP tensors |
|
||||
|------|--------------|---------|-------------|
|
||||
| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 |
|
||||
| `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 |
|
||||
|
||||
The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:
|
||||
|
||||
```
|
||||
llama-server --model <HauhauCS-Q8_K_P.gguf> \
|
||||
--spec-draft-model <MTP-only-draft.gguf> \
|
||||
--spec-type draft-mtp \
|
||||
--spec-draft-n-max 2
|
||||
```
|
||||
|
||||
Compatible MTP-only draft GGUFs on Hugging Face:
|
||||
|
||||
| Repo | Notes |
|
||||
|------|-------|
|
||||
| `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` |
|
||||
| `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` |
|
||||
| `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) |
|
||||
| `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) |
|
||||
|
||||
Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.
|
||||
|
||||
On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
|
||||
|
||||
## Draft Models
|
||||
|
||||
DFlash draft model sizes for the models on these servers:
|
||||
|
||||
| Target model | DFlash draft | Draft size |
|
||||
|--------------|--------------|------------|
|
||||
| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B |
|
||||
| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B |
|
||||
| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B |
|
||||
| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B |
|
||||
|
||||
The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is retired (2026-08-16) and its unit disabled (2026-08-17). The RTX 3090 now runs the PP worker for the Dankclaw head (see [[Qwen3.8 PP Deployment]]). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
- FP8 KV cache is valid on CUDA 12.
|
||||
- FP8 weights compute through a fallback path; no native FP8 tensor cores.
|
||||
- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.
|
||||
|
||||
Working DFlash flags for this server:
|
||||
|
||||
```
|
||||
--spec-method dflash
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
|
||||
--spec-tokens 8
|
||||
--kv-cache-dtype fp8
|
||||
```
|
||||
|
||||
Example DFlash + AWQ on the RTX 3090:
|
||||
|
||||
```
|
||||
docker run --rm --name vllm-qwen36-dflash \
|
||||
--gpus all --ipc=host \
|
||||
-p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
docker.io/vllm/vllm-openai:latest \
|
||||
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
|
||||
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
|
||||
--attention-backend flash_attn \
|
||||
--gpu-memory-utilization 0.95
|
||||
```
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
`frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench.
|
||||
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
|
||||
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
|
||||
- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
|
||||
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
|
||||
- Use `rejection_sample_method` default `strict` for lossless output.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Quantization by Hardware Architecture]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
- [[Dankclaw System Specs]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[VLLM Overview]]
|
||||
- [[deployment]]
|
||||
Reference in new issue
Block a user