diff --git a/AGENTS.md b/AGENTS.md index 2b331d7..135e454 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -174,6 +174,9 @@ These servers serve real users (about 3 people). Running a bench while the service is up starves the GPU and corrupts results. Restart the service when the bench finishes. +ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes. +DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE. + 1. Stop the service before any benchmark: `sudo systemctl stop frank-default-small` (Dankserver) or `sudo systemctl stop frank-default-large` (Dankclaw). diff --git a/docs/Research/Llama Cpp Code Diff.md b/docs/Research/Llama Cpp Code Diff.md index 18aa92b..2083769 100644 --- a/docs/Research/Llama Cpp Code Diff.md +++ b/docs/Research/Llama Cpp Code Diff.md @@ -28,7 +28,7 @@ Where quant lives in each runtime (paths relative to the repo root): | `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode | | `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard | | `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags | -| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` | +| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` | Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha `c49ebdbd`): @@ -107,6 +107,95 @@ MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`. - **Mainline on Strix** works for correctness with standard quants but loses FPX, IU4, TurboQuant, and MTEA. +## 6. KV cache quant table + +Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk` +key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per- +value. Columns mark **origin** (`*` = mainline-provided) and whether the type is +**KV-native** (designed for the cache) or **weight-family reused for KV** (a +model-weight quant the runtime also accepts in `-ctk`/`-ctv`). + +**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the +exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale). +This is the true in-RAM cost per value, so it is what sets KV-cache size and +quantize/dequant throughput. A footnote marks the few types whose **nominal** +figure (older 1-byte-scale layout / data-bits-only) diverges from this exact +number; the table uses the exact number throughout. + +### 6a. All KV-usable types, by bit density (interleaved) + +| bpw | Type | Origin | KV-native | Notes | +|-----|------|--------|-----------|-------| +| 32.0 | f32 | mainline * | yes | full precision | +| 16.0 | f16 | mainline * | yes | default | +| 16.0 | bf16 | mainline * | yes | | +| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version | +| 8.13 | **q8_0** | mainline * | yes | | +| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale | +| 6.63 | iq6_K | ik | weight→KV | IQK family | +| 6.56 | q6_K | mainline * | weight→KV | | +| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) | +| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) | +| 5.50 | q5_K | mainline * | weight→KV | | +| 5.50 | iq5_K | ik | weight→KV | IQK family | +| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) | +| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) | +| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) | +| 4.50 | iq4_K | ik | weight→KV | IQK family | +| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 | +| 4.25 | iq4_ks | ik | weight→KV | IQK subsample | +| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV | +| 4.25 | mxfp4 | ik * | weight→KV | | +| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 | +| 3.44 | iq3_K | ik | weight→KV | IQK family | +| 3.44 | iq3_s | mainline * | weight→KV | | +| 2.69 | iq2_kl | ik | weight→KV | IQK subsample | +| 2.38 | iq2_K | ik | weight→KV | IQK family | +| 2.19 | iq2_ks | ik | weight→KV | IQK subsample | +| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) | +| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) | + +### 6b. Origin legend + +- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp` + `kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked + with `*` and `yes` in the KV-native column. +- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats + (add a third `V` type for values only). Marked `*` here. +- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants + ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the + `IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and + `mxfp4`. ik's `-ctk` accepts a broader set than mainline's. + +### 6c. Trellis, IQK, MXFP4 — where they sit + +- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit + ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest + quality per bit of the family; lowest-density entries in the whole table. +- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44, + `iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 / + `iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family + (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM. +- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry. + +### 6d. Hadamard is a transform, not a row + +`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform +applied on top of a base KV quant**, not its own bit-density row. It spreads +quantization error, improving accuracy for the underlying type; ik's guidance is +**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea — +a packing transform on the K-family — and is likewise a modifier, not a quant. + +### 6e. Server notes + +- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` / + `turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on + Strix is higher-risk (see §4). +- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with + FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is + confirmed for that type. Keep FlashAttention and the KV quant version in lockstep + — they are coupled (a quant change that drops FA off the path changes KV math). + ## 5. Recommendations for our fleet - **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for