Frank
a518fc7c22
Parent-written (scout summaries were corroborated against the files, not trusted): every restatement of an endpoint shape now cites the endpoint page, and four claims contradicted by today's fleet state are corrected. - Quantization Production: the challenger-contract line quoted the retired compaction-hub window; now cites [[frank-zealot]]'s Current shape. The 3090 worked example is era-tagged. Sources listed [[frank-dragoon]] three times. - Quantization by Hardware Architecture: said the 3090 lives on Dankserver (it moved to Dankclaw 2026-09-16; the Eras table owns the move) and ranked the retired Ling-3.0 Quark INT4 service as the ACTIVE Dankclaw workload. The active service is [[frank-dragoon]] on a closed `.hgn` format. - GGUF Compatibility: Related claimed the ftype-115 ROCmFPX file was "in production"; that is an era now, so it points at [[kingjones-q8agent-rocmfpx]]. - Model Swap and Rollback: dropped the fragment byte-count ledger (AGENTS.md owns it), the Ampere FP8 hardware claim ([[Quantization by Hardware Architecture]] owns it), halogen's HTTP surface ([[frank-dragoon]]'s wire contract) and three brittle unit-file line refs (accepted loss: line numbers are derivable, the root-unit law stays). Fixed the per-slot example, which described the retired 3-slot arithmetic, and a Related bullet that called [[frank-archon]] retired while linking the same page twice. Census over the diff: 1 orphan line, the deleted unit line-number refs above.
146 lines
5.2 KiB
Markdown
146 lines
5.2 KiB
Markdown
---
|
|
type: reference
|
|
description: "This page maps LLM quantization formats to the GPU and CPU architectures that support them."
|
|
tags:
|
|
- llm
|
|
- quantization
|
|
timestamp: 2026-08-28
|
|
---
|
|
|
|
# Quantization by Hardware Architecture
|
|
|
|
## Overview
|
|
|
|
This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?
|
|
|
|
Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:
|
|
|
|
- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
|
|
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
|
|
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
|
|
- AMD CDNA (MI300 series): FP8 or MXFP4.
|
|
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
|
|
- CPU-only: any GGUF quant.
|
|
|
|
## Quantization Types
|
|
|
|
### FP8 (E4M3)
|
|
|
|
FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).
|
|
|
|
| Hardware | Native support | Notes |
|
|
|----------|----------------|-------|
|
|
| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores |
|
|
| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores |
|
|
| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback |
|
|
| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only |
|
|
| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 |
|
|
| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER |
|
|
| AMD MI250/MI325 | no | simulation: dequantize to FP16 |
|
|
| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only |
|
|
| CPU | no | FP8 models run via dequant |
|
|
|
|
Rules:
|
|
|
|
- Use FP8 on Ada, Hopper, Blackwell, or MI300+.
|
|
- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
|
|
- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.
|
|
|
|
### INT8
|
|
|
|
INT8 W8A8 is broadly supported.
|
|
|
|
| Hardware | Native support |
|
|
|----------|----------------|
|
|
| NVIDIA Volta through Blackwell | yes |
|
|
| AMD MI300/MI350 | yes |
|
|
| AMD RDNA 3/3.5 | no |
|
|
| x86 CPU | yes, via llama.cpp or llm-compressor |
|
|
|
|
### INT4 (AWQ, GPTQ)
|
|
|
|
INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.
|
|
|
|
| Hardware | AWQ | GPTQ |
|
|
|----------|-----|------|
|
|
| NVIDIA Turing | yes | yes |
|
|
| NVIDIA Ampere | yes | yes |
|
|
| NVIDIA Ada / Hopper / Blackwell | yes | yes |
|
|
| AMD ROCm (vLLM) | no | no |
|
|
| AMD ROCm (Quark) | yes | yes |
|
|
| AMD ROCm GGUF | yes | yes |
|
|
|
|
Rules:
|
|
|
|
- AWQ is the default choice for NVIDIA with vLLM.
|
|
- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.
|
|
|
|
### GGUF (Q4_0, Q5_K_M, Q8_0, etc.)
|
|
|
|
GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.
|
|
|
|
Good GGUF quant choices:
|
|
|
|
| Use case | Quant |
|
|
|----------|-------|
|
|
| Quality first | Q8_0, Q6_K |
|
|
| Balanced | Q5_K_M, Q4_K_M |
|
|
| Small footprint | Q4_0, IQ4_XS |
|
|
| Smallest | IQ2_XS, Q2_K |
|
|
|
|
### FP4
|
|
|
|
FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).
|
|
|
|
| Hardware | NVFP4 | MXFP4 |
|
|
|----------|-------|-------|
|
|
| NVIDIA Blackwell | native | aligned |
|
|
| NVIDIA Ada / Ampere | emulation | emulation |
|
|
| AMD MI300X/MI355 | via ROCm | native |
|
|
| AMD MI250/MI325 | simulation | simulation |
|
|
| AMD RDNA 3 | emulation | aligned, beta |
|
|
|
|
### ROCmFP4 (ROCmFPX fork)
|
|
|
|
ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.
|
|
|
|
Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.
|
|
|
|
## Choices for This Project
|
|
|
|
### The RTX 3090 (24 GB, Ampere) — frank-zealot's card
|
|
|
|
The 3090 has no native FP8 or FP4 tensor cores. It has sat in Dankclaw since
|
|
2026-09-16; [[frank-zealot]]'s Eras table records the move off Dankserver. Use:
|
|
|
|
| Ranking | Format | Why |
|
|
|---------|--------|-----|
|
|
| 1 | AWQ 4-bit | best fit for 24 GB; native kernels |
|
|
| 2 | GPTQ 4-bit | good alternative to AWQ |
|
|
| 3 | BF16 + KV offload | quality reference; CPU-offload experts |
|
|
| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win |
|
|
|
|
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
|
|
|
RDNA 3.5 has no native FP8 tensor cores and the APU shares its memory, so the
|
|
GPU budget is the GTT aperture and every live figure on the box is
|
|
[[Dankclaw System Specs]]. What serves here today is [[frank-dragoon]]: a
|
|
vendor `.hgn` checkpoint, a format this page cannot rank because no public
|
|
quantizer exists for it ([[Quantization Production]]).
|
|
|
|
| Ranking | Format | Why |
|
|
|---------|--------|-----|
|
|
| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed |
|
|
| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) |
|
|
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] |
|
|
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
|
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
|
- [[Dankclaw System Specs]]
|
|
- [[Dankserver System Specs]]
|
|
- [[VLLM Overview]]
|
|
- [[Shared Memory Architecture Dankclaw]] |