docs(kv): add bit-density KV cache quant table across 3 runtimes
- Place ik's Trellis (IQ*_KT), IQK (IQ*_K), MXFP4, and dedicated fast KV type Q8_KV in a single high->low bits-per-value table with ROCmFPX TurboQuant KV. - Mark mainline-provided types (*), KV-native vs weight-family-reused-for-KV. - Hadamard annotated as a transform on top of a base quant, not a row. - bpw read from exact ggml block structs (QK_K=256, fp16 scale): type_size x 8 / QK, verified against the struct comment "6.5625 bits per weight" and the trait table. Notes where legacy nominal figures (4.0/4.25) diverge. - AGENTS.md: add benchmark directive (use llama-bench / vllm bench; ask before rolling your own).
This commit is contained in:
Frank
@@ -28,7 +28,7 @@ Where quant lives in each runtime (paths relative to the repo root):
|
||||
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
|
||||
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
|
||||
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
|
||||
| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` |
|
||||
| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |
|
||||
|
||||
Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
|
||||
`c49ebdbd`):
|
||||
@@ -107,6 +107,95 @@ MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
|
||||
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
|
||||
IU4, TurboQuant, and MTEA.
|
||||
|
||||
## 6. KV cache quant table
|
||||
|
||||
Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
|
||||
key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
|
||||
value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
|
||||
**KV-native** (designed for the cache) or **weight-family reused for KV** (a
|
||||
model-weight quant the runtime also accepts in `-ctk`/`-ctv`).
|
||||
|
||||
**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
|
||||
exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
|
||||
This is the true in-RAM cost per value, so it is what sets KV-cache size and
|
||||
quantize/dequant throughput. A footnote marks the few types whose **nominal**
|
||||
figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
|
||||
number; the table uses the exact number throughout.
|
||||
|
||||
### 6a. All KV-usable types, by bit density (interleaved)
|
||||
|
||||
| bpw | Type | Origin | KV-native | Notes |
|
||||
|-----|------|--------|-----------|-------|
|
||||
| 32.0 | f32 | mainline * | yes | full precision |
|
||||
| 16.0 | f16 | mainline * | yes | default |
|
||||
| 16.0 | bf16 | mainline * | yes | |
|
||||
| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
|
||||
| 8.13 | **q8_0** | mainline * | yes | |
|
||||
| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
|
||||
| 6.63 | iq6_K | ik | weight→KV | IQK family |
|
||||
| 6.56 | q6_K | mainline * | weight→KV | |
|
||||
| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
|
||||
| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
|
||||
| 5.50 | q5_K | mainline * | weight→KV | |
|
||||
| 5.50 | iq5_K | ik | weight→KV | IQK family |
|
||||
| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
|
||||
| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
|
||||
| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
|
||||
| 4.50 | iq4_K | ik | weight→KV | IQK family |
|
||||
| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
|
||||
| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
|
||||
| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
|
||||
| 4.25 | mxfp4 | ik * | weight→KV | |
|
||||
| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
|
||||
| 3.44 | iq3_K | ik | weight→KV | IQK family |
|
||||
| 3.44 | iq3_s | mainline * | weight→KV | |
|
||||
| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
|
||||
| 2.38 | iq2_K | ik | weight→KV | IQK family |
|
||||
| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
|
||||
| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
|
||||
| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |
|
||||
|
||||
### 6b. Origin legend
|
||||
|
||||
- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
|
||||
`kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
|
||||
with `*` and `yes` in the KV-native column.
|
||||
- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
|
||||
(add a third `V` type for values only). Marked `*` here.
|
||||
- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
|
||||
ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
|
||||
`IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
|
||||
`mxfp4`. ik's `-ctk` accepts a broader set than mainline's.
|
||||
|
||||
### 6c. Trellis, IQK, MXFP4 — where they sit
|
||||
|
||||
- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
|
||||
ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
|
||||
quality per bit of the family; lowest-density entries in the whole table.
|
||||
- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
|
||||
`iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
|
||||
`iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
|
||||
(iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
|
||||
- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.
|
||||
|
||||
### 6d. Hadamard is a transform, not a row
|
||||
|
||||
`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
|
||||
applied on top of a base KV quant**, not its own bit-density row. It spreads
|
||||
quantization error, improving accuracy for the underlying type; ik's guidance is
|
||||
**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
|
||||
a packing transform on the K-family — and is likewise a modifier, not a quant.
|
||||
|
||||
### 6e. Server notes
|
||||
|
||||
- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
|
||||
`turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
|
||||
Strix is higher-risk (see §4).
|
||||
- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
|
||||
FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
|
||||
confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
|
||||
— they are coupled (a quant change that drops FA off the path changes KV math).
|
||||
|
||||
## 5. Recommendations for our fleet
|
||||
|
||||
- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
|
||||
|
||||
Reference in New Issue
Block a user