Frank
993b6af576
- DFlash DSpark Speculative Decoding.md: verified method details (DFlash block diffusion from Z Lab arXiv:2602.06036; DSpark semi-autoregressive from DeepSeek arXiv:2607.05147), runtime support matrix (vLLM/llama.cpp/SGLang), exact vLLM --speculative-config recipes (incl. dflash/dspark), llama.cpp --spec-type recipes with sidecar auto-discovery, draft model table (z-lab Qwen3.6-27B-DFlash 2B, Qwen3.6-35B-A3B-DFlash 0.4B), deployment guidance for RTX 3090 (AWQ+Dflash) and Strix Halo (ROCmFPX MTP), measured 45-50 t/s MTP decode on Strix Halo - Quantization by Hardware Architecture.md: FP8/INT8/INT4/GPTQ/AWQ/GGUF/FP4/ ROCmFP4 support per NVIDIA (Turing/Ampere/Ada/Hopper/Blackwell), AMD (CDNA MI250/MI300/MI350, RDNA 3/3.5/4), and CPU; recommended picks per Dankserver (AWQ/GPTQ 4-bit) and Dankclaw (ROCmFP4/Q4_K_M) - index.md: link both new pages under Speculative Decoding & Quantization
4.4 KiB
Quantization by Hardware Architecture
Overview
This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?
Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:
- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
- AMD CDNA (MI300 series): FP8 or MXFP4.
- AMD RDNA 3 / Strix Halo (gfx11): GGUF Q4_K_M or ROCmFP4.
- CPU-only: any GGUF quant.
Quantization Types
FP8 (E4M3)
FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).
| Hardware | Native support | Notes |
|---|---|---|
| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores |
| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores |
| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback |
| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only |
| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 |
| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER |
| AMD MI250/MI325 | no | simulation: dequantize to FP16 |
| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only |
| CPU | no | FP8 models run via dequant |
Rules:
- Use FP8 on Ada, Hopper, Blackwell, or MI300+.
- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.
INT8
INT8 W8A8 is broadly supported.
| Hardware | Native support |
|---|---|
| NVIDIA Volta through Blackwell | yes |
| AMD MI300/MI350 | yes |
| AMD RDNA 3/3.5 | no |
| x86 CPU | yes, via llama.cpp or llm-compressor |
INT4 (AWQ, GPTQ)
INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.
| Hardware | AWQ | GPTQ |
|---|---|---|
| NVIDIA Turing | yes | yes |
| NVIDIA Ampere | yes | yes |
| NVIDIA Ada / Hopper / Blackwell | yes | yes |
| AMD ROCm (vLLM) | no | no |
| AMD ROCm (Quark) | yes | yes |
| AMD ROCm GGUF | yes | yes |
Rules:
- AWQ is the default choice for NVIDIA with vLLM.
- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.
GGUF (Q4_0, Q5_K_M, Q8_0, etc.)
GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.
Good GGUF quant choices:
| Use case | Quant |
|---|---|
| Quality first | Q8_0, Q6_K |
| Balanced | Q5_K_M, Q4_K_M |
| Small footprint | Q4_0, IQ4_XS |
| Smallest | IQ2_XS, Q2_K |
FP4
FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).
| Hardware | NVFP4 | MXFP4 |
|---|---|---|
| NVIDIA Blackwell | native | aligned |
| NVIDIA Ada / Ampere | emulation | emulation |
| AMD MI300X/MI355 | via ROCm | native |
| AMD MI250/MI325 | simulation | simulation |
| AMD RDNA 3 | emulation | aligned, beta |
ROCmFP4 (ROCmFPX fork)
ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.
Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.
Choices for This Project
Dankserver (RTX 3090, 24 GB, Ampere)
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
| Ranking | Format | Why |
|---|---|---|
| 1 | AWQ 4-bit | best fit for 24 GB; native kernels |
| 2 | GPTQ 4-bit | good alternative to AWQ |
| 3 | BF16 + KV offload | quality reference; CPU-offload experts |
| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win |
Dankclaw (Strix Halo gfx1151, RDNA 3.5)
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. Use GGUF through llama.cpp, or ROCmFPX formats:
| Ranking | Format | Why |
|---|---|---|
| 1 | ROCmFP4 (ROCmFPX fork) | native ROCm kernels; proven with Ling-3.0-flash MTP |
| 2 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp |
| 3 | FP8 (vLLM) | software emulation; memory savings only |