docs(kv): add bit-density KV cache quant table across 3 runtimes

- Place ik's Trellis (IQ*_KT), IQK (IQ*_K), MXFP4, and dedicated fast KV type Q8_KV in a single high->low bits-per-value table with ROCmFPX TurboQuant KV. - Mark mainline-provided types (*), KV-native vs weight-family-reused-for-KV. - Hadamard annotated as a transform on top of a base quant, not a row. - bpw read from exact ggml block structs (QK_K=256, fp16 scale): type_size x 8 / QK, verified against the struct comment "6.5625 bits per weight" and the trait table. Notes where legacy nominal figures (4.0/4.25) diverge. - AGENTS.md: add benchmark directive (use llama-bench / vllm bench; ask before rolling your own).
2026-08-25 20:03:13 -07:00
parent 4308f62004
commit 2277a2e9ac
2 changed files with 93 additions and 1 deletions
@@ -174,6 +174,9 @@ These servers serve real users (about 3 people). Running a bench while the
 service is up starves the GPU and corrupts results. Restart the service when
 the bench finishes.

+ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
+DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
+
 1. Stop the service before any benchmark:
   `sudo systemctl stop frank-default-small` (Dankserver) or
   `sudo systemctl stop frank-default-large` (Dankclaw).
@@ -28,7 +28,7 @@ Where quant lives in each runtime (paths relative to the repo root):
 | `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
 | `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
 | `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
-| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` |
+| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |

 Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
 `c49ebdbd`):
@@ -107,6 +107,95 @@ MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
 - **Mainline on Strix** works for correctness with standard quants but loses FPX,
  IU4, TurboQuant, and MTEA.

+## 6. KV cache quant table
+
+Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
+key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
+value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
+**KV-native** (designed for the cache) or **weight-family reused for KV** (a
+model-weight quant the runtime also accepts in `-ctk`/`-ctv`).
+
+**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
+exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
+This is the true in-RAM cost per value, so it is what sets KV-cache size and
+quantize/dequant throughput. A footnote marks the few types whose **nominal**
+figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
+number; the table uses the exact number throughout.
+
+### 6a. All KV-usable types, by bit density (interleaved)
+
+| bpw | Type | Origin | KV-native | Notes |
+|-----|------|--------|-----------|-------|
+| 32.0 | f32 | mainline * | yes | full precision |
+| 16.0 | f16 | mainline * | yes | default |
+| 16.0 | bf16 | mainline * | yes | |
+| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
+| 8.13 | **q8_0** | mainline * | yes | |
+| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
+| 6.63 | iq6_K | ik | weight→KV | IQK family |
+| 6.56 | q6_K | mainline * | weight→KV | |
+| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
+| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
+| 5.50 | q5_K | mainline * | weight→KV | |
+| 5.50 | iq5_K | ik | weight→KV | IQK family |
+| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
+| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
+| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
+| 4.50 | iq4_K | ik | weight→KV | IQK family |
+| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
+| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
+| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
+| 4.25 | mxfp4 | ik * | weight→KV | |
+| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
+| 3.44 | iq3_K | ik | weight→KV | IQK family |
+| 3.44 | iq3_s | mainline * | weight→KV | |
+| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
+| 2.38 | iq2_K | ik | weight→KV | IQK family |
+| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
+| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
+| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |
+
+### 6b. Origin legend
+
+- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
+  `kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
+  with `*` and `yes` in the KV-native column.
+- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
+  (add a third `V` type for values only). Marked `*` here.
+- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
+  ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
+  `IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
+  `mxfp4`. ik's `-ctk` accepts a broader set than mainline's.
+
+### 6c. Trellis, IQK, MXFP4 — where they sit
+
+- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
+  ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
+  quality per bit of the family; lowest-density entries in the whole table.
+- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
+  `iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
+  `iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
+  (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
+- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.
+
+### 6d. Hadamard is a transform, not a row
+
+`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
+applied on top of a base KV quant**, not its own bit-density row. It spreads
+quantization error, improving accuracy for the underlying type; ik's guidance is
+**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
+a packing transform on the K-family — and is likewise a modifier, not a quant.
+
+### 6e. Server notes
+
+- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
+  `turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
+  Strix is higher-risk (see §4).
+- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
+  FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
+  confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
+  — they are coupled (a quant change that drops FA off the path changes KV math).
+
 ## 5. Recommendations for our fleet

 - **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for