dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add DFlash/DSpark speculative decoding research + quantization-by-hardware guide

Browse Source 
- DFlash DSpark Speculative Decoding.md: verified method details (DFlash
  block diffusion from Z Lab arXiv:2602.06036; DSpark semi-autoregressive from
  DeepSeek arXiv:2607.05147), runtime support matrix (vLLM/llama.cpp/SGLang),
  exact vLLM --speculative-config recipes (incl. dflash/dspark), llama.cpp
  --spec-type recipes with sidecar auto-discovery, draft model table
  (z-lab Qwen3.6-27B-DFlash 2B, Qwen3.6-35B-A3B-DFlash 0.4B), deployment
  guidance for RTX 3090 (AWQ+Dflash) and Strix Halo (ROCmFPX MTP), measured
  45-50 t/s MTP decode on Strix Halo
- Quantization by Hardware Architecture.md: FP8/INT8/INT4/GPTQ/AWQ/GGUF/FP4/
  ROCmFP4 support per NVIDIA (Turing/Ampere/Ada/Hopper/Blackwell), AMD
  (CDNA MI250/MI300/MI350, RDNA 3/3.5/4), and CPU; recommended picks per
  Dankserver (AWQ/GPTQ 4-bit) and Dankclaw (ROCmFP4/Q4_K_M)
- index.md: link both new pages under Speculative Decoding & Quantization
This commit is contained in:
Frank
2026-08-10 21:09:17 -07:00
parent 90c4883f89
commit 993b6af576
3 changed files with 317 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DFlash DSpark Speculative Decoding.md
+181
View File
@@ -0,0 +1,181 @@
# DFlash and DSpark Speculative Decoding
## Overview
DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.
DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.
## How DFlash Works
DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:
- The draft model reuses the embedding and LM head from the target model.
- Only the intermediate layers are trained.
- It fuses target-model hidden features into the KV projections of every draft layer.
- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.
Results from the DFlash paper (arXiv:2602.06036):
- Over 6x lossless speedup on Qwen3-8B.
- Up to 2.5x higher speedup than EAGLE-3.
- Diffusion drafting cost is flat, so block size does not increase cost.
## How DSpark Works
DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.
Results from the DSpark paper:
- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
- Useful when parallel drafts decay in acceptance rate.
Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.
## Runtime Support
| Runtime | DFlash | DSpark | MTP |
|---------|--------|--------|-----|
| vLLM | yes (v0.20.1+) | yes | yes |
| llama.cpp | yes | yes | yes |
| SGLang | yes | no | yes |
### vLLM
vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object.
DFlash on vLLM:
```
vllm serve Qwen/Qwen3.6-35B-A3B \
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
--attention-backend flash_attn \
--max-num-batched-tokens 32768
```
Legacy flags also work:
```
vllm serve Qwen/Qwen3.6-35B-A3B \
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8
```
DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.
DSpark on vLLM:
```
vllm serve <target-model> \
--speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'
```
### llama.cpp
llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.
DFlash on llama.cpp:
```
llama-server --hf-repo <target-GGUF> \
--spec-draft-hf <draft-GGUF> \
--spec-type draft-dflash \
--spec-draft-n-max 16
```
DSpark on llama.cpp:
```
llama-server --hf-repo <target-GGUF> \
--spec-draft-hf <draft-GGUF> \
--spec-type draft-dspark
```
MTP on llama.cpp:
```
llama-server --hf-repo <target-with-MTP> \
--spec-type draft-mtp \
--spec-draft-n-max 2
```
## Draft Models
DFlash draft model sizes for the models on these servers:
| Target model | DFlash draft | Draft size |
|--------------|--------------|------------|
| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B |
| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B |
| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B |
| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B |
The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.
## Deployment on Dankserver (NVIDIA RTX 3090)
Dankserver runs vLLM in a Podman container with the RTX 3090. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
- FP8 KV cache is valid on CUDA 12.
- FP8 weights compute through a fallback path; no native FP8 tensor cores.
- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.
Working DFlash flags for this server:
```
--spec-method dflash
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
--spec-tokens 8
--kv-cache-dtype fp8
```
Example DFlash + AWQ on the RTX 3090:
```
docker run --rm --name vllm-qwen36-dflash \
--gpus all --ipc=host \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
docker.io/vllm/vllm-openai:latest \
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
--attention-backend flash_attn \
--gpu-memory-utilization 0.95
```
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results:
- 45-50 t/s decode with MTP speculation.
- ~37 t/s plain decode.
- ~95% draft acceptance with `--spec-draft-n-max 2`.
ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp.
Ling-3.0-flash-related facts:
- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active.
- The MTP (NextN) head is preserved in the GGUF conversion.
- The raulvidis GGUF repo targets Strix Halo (gfx1151).
If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151.
## Notes
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
- Use `rejection_sample_method` default `strict` for lossless output.
## See Also
- [[Quantization by Hardware Architecture]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
docs/Quantization by Hardware Architecture.md
+131
View File
@@ -0,0 +1,131 @@
# Quantization by Hardware Architecture
## Overview
This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?
Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:
- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
- AMD CDNA (MI300 series): FP8 or MXFP4.
- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4.
- CPU-only: any GGUF quant.
## Quantization Types
### FP8 (E4M3)
FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).
| Hardware | Native support | Notes |
|----------|----------------|-------|
| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores |
| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores |
| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback |
| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only |
| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 |
| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER |
| AMD MI250/MI325 | no | simulation: dequantize to FP16 |
| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only |
| CPU | no | FP8 models run via dequant |
Rules:
- Use FP8 on Ada, Hopper, Blackwell, or MI300+.
- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.
### INT8
INT8 W8A8 is broadly supported.
| Hardware | Native support |
|----------|----------------|
| NVIDIA Volta through Blackwell | yes |
| AMD MI300/MI350 | yes |
| AMD RDNA 3/3.5 | no |
| x86 CPU | yes, via llama.cpp or llm-compressor |
### INT4 (AWQ, GPTQ)
INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.
| Hardware | AWQ | GPTQ |
|----------|-----|------|
| NVIDIA Turing | yes | yes |
| NVIDIA Ampere | yes | yes |
| NVIDIA Ada / Hopper / Blackwell | yes | yes |
| AMD ROCm (vLLM) | no | no |
| AMD ROCm (Quark) | yes | yes |
| AMD ROCm GGUF | yes | yes |
Rules:
- AWQ is the default choice for NVIDIA with vLLM.
- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.
### GGUF (Q4_0, Q5_K_M, Q8_0, etc.)
GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.
Good GGUF quant choices:
| Use case | Quant |
|----------|-------|
| Quality first | Q8_0, Q6_K |
| Balanced | Q5_K_M, Q4_K_M |
| Small footprint | Q4_0, IQ4_XS |
| Smallest | IQ2_XS, Q2_K |
### FP4
FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).
| Hardware | NVFP4 | MXFP4 |
|----------|-------|-------|
| NVIDIA Blackwell | native | aligned |
| NVIDIA Ada / Ampere | emulation | emulation |
| AMD MI300X/MI355 | via ROCm | native |
| AMD MI250/MI325 | simulation | simulation |
| AMD RDNA 3 | emulation | aligned, beta |
### ROCmFP4 (ROCmFPX fork)
ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.
Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.
## Choices for This Project
### Dankserver (RTX 3090, 24 GB, Ampere)
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
| Ranking | Format | Why |
|---------|--------|-----|
| 1 | AWQ 4-bit | best fit for 24 GB; native kernels |
| 2 | GPTQ 4-bit | good alternative to AWQ |
| 3 | BF16 + KV offload | quality reference; CPU-offload experts |
| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win |
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats:
| Ranking | Format | Why |
|---------|--------|-----|
| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP |
| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp |
| 3 | FP8 (vLLM) | software emulation; memory savings only |
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[Shared Memory Architecture Dankclaw]]
docs/index.md
+5
View File
@@ -47,3 +47,8 @@ graph LR
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
## Speculative Decoding & Quantization
- [[DFlash DSpark Speculative Decoding]] — block diffusion (DFlash) and semi-autoregressive (DSpark) drafting on vLLM and llama.cpp
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU