diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md new file mode 100644 index 0000000..940fe09 --- /dev/null +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -0,0 +1,181 @@ +# DFlash and DSpark Speculative Decoding + +## Overview + +DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads. + +DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run. + +## How DFlash Works + +DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points: + +- The draft model reuses the embedding and LM head from the target model. +- Only the intermediate layers are trained. +- It fuses target-model hidden features into the KV projections of every draft layer. +- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens. + +Results from the DFlash paper (arXiv:2602.06036): + +- Over 6x lossless speedup on Qwen3-8B. +- Up to 2.5x higher speedup than EAGLE-3. +- Diffusion drafting cost is flat, so block size does not increase cost. + +## How DSpark Works + +DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk. + +Results from the DSpark paper: + +- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic. +- Useful when parallel drafts decay in acceptance rate. + +Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler. + +## Runtime Support + +| Runtime | DFlash | DSpark | MTP | +|---------|--------|--------|-----| +| vLLM | yes (v0.20.1+) | yes | yes | +| llama.cpp | yes | yes | yes | +| SGLang | yes | no | yes | + +### vLLM + +vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object. + +DFlash on vLLM: + +``` +vllm serve Qwen/Qwen3.6-35B-A3B \ + --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \ + --attention-backend flash_attn \ + --max-num-batched-tokens 32768 +``` + +Legacy flags also work: + +``` +vllm serve Qwen/Qwen3.6-35B-A3B \ + --spec-method dflash \ + --spec-model z-lab/Qwen3.6-35B-A3B-DFlash \ + --spec-tokens 8 +``` + +DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash. + +DSpark on vLLM: + +``` +vllm serve \ + --speculative-config '{"method": "dspark", "model": "", "num_speculative_tokens": 15}' +``` + +### llama.cpp + +llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP. + +DFlash on llama.cpp: + +``` +llama-server --hf-repo \ + --spec-draft-hf \ + --spec-type draft-dflash \ + --spec-draft-n-max 16 +``` + +DSpark on llama.cpp: + +``` +llama-server --hf-repo \ + --spec-draft-hf \ + --spec-type draft-dspark +``` + +MTP on llama.cpp: + +``` +llama-server --hf-repo \ + --spec-type draft-mtp \ + --spec-draft-n-max 2 +``` + +## Draft Models + +DFlash draft model sizes for the models on these servers: + +| Target model | DFlash draft | Draft size | +|--------------|--------------|------------| +| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B | +| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B | +| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B | +| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B | + +The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput. + +## Deployment on Dankserver (NVIDIA RTX 3090) + +Dankserver runs vLLM in a Podman container with the RTX 3090. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions: + +- FP8 KV cache is valid on CUDA 12. +- FP8 weights compute through a fallback path; no native FP8 tensor cores. +- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target. + +Working DFlash flags for this server: + +``` +--spec-method dflash +--spec-model z-lab/Qwen3.6-35B-A3B-DFlash +--spec-tokens 8 +--kv-cache-dtype fp8 +``` + +Example DFlash + AWQ on the RTX 3090: + +``` +docker run --rm --name vllm-qwen36-dflash \ + --gpus all --ipc=host \ + -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + docker.io/vllm/vllm-openai:latest \ + cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \ + --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \ + --attention-backend flash_attn \ + --gpu-memory-utilization 0.95 +``` + +## Deployment on Dankclaw (AMD Strix Halo gfx1151) + +Dankclaw runs the ROCmFPX llama.cpp runtime as its primary service. The active service uses the Ling-3.0-flash ROCmFP4 MTP model with `--spec-type draft-mtp`. This gives measured results: + +- 45-50 t/s decode with MTP speculation. +- ~37 t/s plain decode. +- ~95% draft acceptance with `--spec-draft-n-max 2`. + +ROCmFPX is a llama.cpp fork by charlie12345. It adds ROCmFP4 and ROCmFP2 GGUF weight formats with HIP/ROCm and Vulkan kernels. It is not merged into upstream llama.cpp. + +Ling-3.0-flash-related facts: + +- Base model: inclusionAI/Ling-3.0-flash, 124B total / 5.1B active. +- The MTP (NextN) head is preserved in the GGUF conversion. +- The raulvidis GGUF repo targets Strix Halo (gfx1151). + +If you want DFlash on Dankclaw, use a DFlash GGUF draft. Community conversions exist, for example Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp. The ROCm path for DFlash is newer than the MTP path and less proven on gfx1151. + +## Notes + +- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS. +- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. +- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`. +- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist. +- Use `rejection_sample_method` default `strict` for lossless output. + +## See Also + +- [[Quantization by Hardware Architecture]] +- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] +- [[Running Qwen3.6 27B FP8 on Dankclaw]] +- [[Dankclaw System Specs]] +- [[Dankserver System Specs]] +- [[VLLM Overview]] +- [[deployment]] \ No newline at end of file diff --git a/docs/Quantization by Hardware Architecture.md b/docs/Quantization by Hardware Architecture.md new file mode 100644 index 0000000..7122a65 --- /dev/null +++ b/docs/Quantization by Hardware Architecture.md @@ -0,0 +1,131 @@ +# Quantization by Hardware Architecture + +## Overview + +This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware? + +Use this page when you choose a model format for Dankserver or Dankclaw. The short answer: + +- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit. +- NVIDIA Ada (RTX 40 series): FP8 or AWQ. +- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4. +- AMD CDNA (MI300 series): FP8 or MXFP4. +- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4. +- CPU-only: any GGUF quant. + +## Quantization Types + +### FP8 (E4M3) + +FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only). + +| Hardware | Native support | Notes | +|----------|----------------|-------| +| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores | +| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores | +| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback | +| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only | +| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 | +| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER | +| AMD MI250/MI325 | no | simulation: dequantize to FP16 | +| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only | +| CPU | no | FP8 models run via dequant | + +Rules: + +- Use FP8 on Ada, Hopper, Blackwell, or MI300+. +- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead. +- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090. + +### INT8 + +INT8 W8A8 is broadly supported. + +| Hardware | Native support | +|----------|----------------| +| NVIDIA Volta through Blackwell | yes | +| AMD MI300/MI350 | yes | +| AMD RDNA 3/3.5 | no | +| x86 CPU | yes, via llama.cpp or llm-compressor | + +### INT4 (AWQ, GPTQ) + +INT4 is the standard choice for 24 GB GPUs such as the RTX 3090. + +| Hardware | AWQ | GPTQ | +|----------|-----|------| +| NVIDIA Turing | yes | yes | +| NVIDIA Ampere | yes | yes | +| NVIDIA Ada / Hopper / Blackwell | yes | yes | +| AMD ROCm (vLLM) | no | no | +| AMD ROCm (Quark) | yes | yes | +| AMD ROCm GGUF | yes | yes | + +Rules: + +- AWQ is the default choice for NVIDIA with vLLM. +- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix. + +### GGUF (Q4_0, Q5_K_M, Q8_0, etc.) + +GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses. + +Good GGUF quant choices: + +| Use case | Quant | +|----------|-------| +| Quality first | Q8_0, Q6_K | +| Balanced | Q5_K_M, Q4_K_M | +| Small footprint | Q4_0, IQ4_XS | +| Smallest | IQ2_XS, Q2_K | + +### FP4 + +FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard). + +| Hardware | NVFP4 | MXFP4 | +|----------|-------|-------| +| NVIDIA Blackwell | native | aligned | +| NVIDIA Ada / Ampere | emulation | emulation | +| AMD MI300X/MI355 | via ROCm | native | +| AMD MI250/MI325 | simulation | simulation | +| AMD RDNA 3 | emulation | aligned, beta | + +### ROCmFP4 (ROCmFPX fork) + +ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels. + +Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files. + +## Choices for This Project + +### Dankserver (RTX 3090, 24 GB, Ampere) + +The RTX 3090 has no native FP8 or FP4 tensor cores. Use: + +| Ranking | Format | Why | +|---------|--------|-----| +| 1 | AWQ 4-bit | best fit for 24 GB; native kernels | +| 2 | GPTQ 4-bit | good alternative to AWQ | +| 3 | BF16 + KV offload | quality reference; CPU-offload experts | +| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win | + +### Dankclaw (Strix Halo gfx1151, RDNA 3.5) + +RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats: + +| Ranking | Format | Why | +|---------|--------|-----| +| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP | +| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp | +| 3 | FP8 (vLLM) | software emulation; memory savings only | + +## See Also + +- [[DFlash DSpark Speculative Decoding]] +- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] +- [[Running Qwen3.6 27B FP8 on Dankclaw]] +- [[Dankclaw System Specs]] +- [[Dankserver System Specs]] +- [[VLLM Overview]] +- [[Shared Memory Architecture Dankclaw]] \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index 8df746d..356fc78 100644 --- a/docs/index.md +++ b/docs/index.md @@ -47,3 +47,8 @@ graph LR - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU + +## Speculative Decoding & Quantization + +- [[DFlash DSpark Speculative Decoding]] — block diffusion (DFlash) and semi-autoregressive (DSpark) drafting on vLLM and llama.cpp +- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU