Files

T

Frank 327e78b950 docs(vault): destroy the dead weight — 53 pages and the change diary

The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.

2026-10-19 10:56:37 -07:00

5.0 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Quantization by Hardware Architecture

Overview

This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?

Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:

NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
NVIDIA Ada (RTX 40 series): FP8 or AWQ.
NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
AMD CDNA (MI300 series): FP8 or MXFP4.
AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
CPU-only: any GGUF quant.

Quantization Types

FP8 (E4M3)

FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).

Hardware	Native support	Notes
NVIDIA Hopper (SM 9.0)	yes	full FP8 tensor cores
NVIDIA Ada (SM 8.9)	yes	full FP8 tensor cores
NVIDIA Ampere (SM 8.0/8.6)	no	W8A16 via Marlin; W8A8 runs via fallback
NVIDIA Turing (SM 7.5)	no	W8A16 via Marlin only
NVIDIA Blackwell (SM 12.0)	yes	FP8 and native FP4
AMD MI300/MI350 (gfx942/gfx950)	yes	via ROCm AITER
AMD MI250/MI325	no	simulation: dequantize to FP16
AMD RDNA 3/3.5 (gfx11/gfx1151)	no	software emulation only
CPU	no	FP8 models run via dequant

Rules:

Use FP8 on Ada, Hopper, Blackwell, or MI300+.
On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.

INT8

INT8 W8A8 is broadly supported.

Hardware	Native support
NVIDIA Volta through Blackwell	yes
AMD MI300/MI350	yes
AMD RDNA 3/3.5	no
x86 CPU	yes, via llama.cpp or llm-compressor

INT4 (AWQ, GPTQ)

INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.

Hardware	AWQ	GPTQ
NVIDIA Turing	yes	yes
NVIDIA Ampere	yes	yes
NVIDIA Ada / Hopper / Blackwell	yes	yes
AMD ROCm (vLLM)	no	no
AMD ROCm (Quark)	yes	yes
AMD ROCm GGUF	yes	yes

Rules:

AWQ is the default choice for NVIDIA with vLLM.
On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.

GGUF (Q4_0, Q5_K_M, Q8_0, etc.)

GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.

Good GGUF quant choices:

Use case	Quant
Quality first	Q8_0, Q6_K
Balanced	Q5_K_M, Q4_K_M
Small footprint	Q4_0, IQ4_XS
Smallest	IQ2_XS, Q2_K

FP4

FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).

Hardware	NVFP4	MXFP4
NVIDIA Blackwell	native	aligned
NVIDIA Ada / Ampere	emulation	emulation
AMD MI300X/MI355	via ROCm	native
AMD MI250/MI325	simulation	simulation
AMD RDNA 3	emulation	aligned, beta

ROCmFP4 (ROCmFPX fork)

ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.

Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.

Choices for This Project

The RTX 3090 (24 GB, Ampere) — frank-zealot's card

The 3090 has no native FP8 or FP4 tensor cores. It has sat in Dankclaw since 2026-09-16; frank-zealot's Eras table records the move off Dankserver. Use:

Ranking	Format	Why
1	AWQ 4-bit	best fit for 24 GB; native kernels
2	GPTQ 4-bit	good alternative to AWQ
3	BF16 + KV offload	quality reference; CPU-offload experts
4	FP8	only if the FP8 file is all you have; memory savings, no compute win

Dankclaw (Strix Halo gfx1151, RDNA 3.5)

RDNA 3.5 has no native FP8 tensor cores and the APU shares its memory, so the GPU budget is the GTT aperture and every live figure on the box is Dankclaw System Specs. What serves here today is frank-dragoon: a vendor .hgn checkpoint, a format this page cannot rank because no public quantizer exists for it (Quantization Production).

Ranking	Format	Why
1	`.hgn` (deployed)	the halogen engine's own format; production door is closed
2	GGUF K/I quants, K_P blends	the only self-produced option on this card (Quantization Production)
3	INT4 W4A16 (Quark) / MXFP4 (Quark)	the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer:
4	FP8 (vLLM)	software emulation; memory savings only; no compute win

5.0 KiB Raw Blame History