Frank
5829a0bfe8
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
4.7 KiB
Quantization by Hardware Architecture
Overview
This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?
Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:
- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
- AMD CDNA (MI300 series): FP8 or MXFP4.
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
- CPU-only: any GGUF quant.
Quantization Types
FP8 (E4M3)
FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).
| Hardware | Native support | Notes |
|---|---|---|
| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores |
| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores |
| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback |
| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only |
| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 |
| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER |
| AMD MI250/MI325 | no | simulation: dequantize to FP16 |
| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only |
| CPU | no | FP8 models run via dequant |
Rules:
- Use FP8 on Ada, Hopper, Blackwell, or MI300+.
- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.
INT8
INT8 W8A8 is broadly supported.
| Hardware | Native support |
|---|---|
| NVIDIA Volta through Blackwell | yes |
| AMD MI300/MI350 | yes |
| AMD RDNA 3/3.5 | no |
| x86 CPU | yes, via llama.cpp or llm-compressor |
INT4 (AWQ, GPTQ)
INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.
| Hardware | AWQ | GPTQ |
|---|---|---|
| NVIDIA Turing | yes | yes |
| NVIDIA Ampere | yes | yes |
| NVIDIA Ada / Hopper / Blackwell | yes | yes |
| AMD ROCm (vLLM) | no | no |
| AMD ROCm (Quark) | yes | yes |
| AMD ROCm GGUF | yes | yes |
Rules:
- AWQ is the default choice for NVIDIA with vLLM.
- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.
GGUF (Q4_0, Q5_K_M, Q8_0, etc.)
GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.
Good GGUF quant choices:
| Use case | Quant |
|---|---|
| Quality first | Q8_0, Q6_K |
| Balanced | Q5_K_M, Q4_K_M |
| Small footprint | Q4_0, IQ4_XS |
| Smallest | IQ2_XS, Q2_K |
FP4
FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).
| Hardware | NVFP4 | MXFP4 |
|---|---|---|
| NVIDIA Blackwell | native | aligned |
| NVIDIA Ada / Ampere | emulation | emulation |
| AMD MI300X/MI355 | via ROCm | native |
| AMD MI250/MI325 | simulation | simulation |
| AMD RDNA 3 | emulation | aligned, beta |
ROCmFP4 (ROCmFPX fork)
ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.
Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.
Choices for This Project
Dankserver (RTX 3090, 24 GB, Ampere)
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
| Ranking | Format | Why |
|---|---|---|
| 1 | AWQ 4-bit | best fit for 24 GB; native kernels |
| 2 | GPTQ 4-bit | good alternative to AWQ |
| 3 | BF16 + KV offload | quality reference; CPU-offload experts |
| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win |
Dankclaw (Strix Halo gfx1151, RDNA 3.5)
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
| Ranking | Format | Why |
|---|---|---|
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |