serverAppConfigs/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md

---
type: research
title: Llama.cpp Code Diff
description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
tags:
  - llama.cpp
  - quantization
  - ggml
  - ROCmFPX
  - ik_llama
  - code-diff
timestamp: 2026-09-01
---

# Llama.cpp Code Diff

Code-level diff of the three runtimes, focused on **quantization code** and the
differences that affect our two servers. Positioning overview is in
[[Llama Cpp Runtimes]].

## 1. Quantization file map

Where quant lives in each runtime (paths relative to the repo root):

| File / path | Mainline | ROCmFPX | ik_llama.cpp |
|-------------|----------|---------|--------------|
| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |

Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
`c49ebdbd`):
- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
  `Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
  `Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
  `STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
  FP2=113), `ROCmI4`=118.

## 2. Comparison table

| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |

## 3. Code-difference narrative

**What each fork ADDED that mainline lacks:**
- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
  format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
  `*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
  TurboQuant KV types, MTEA attention kernel. Mainline does not know these
  tensor types. It is a *fork* that adds a new backend: custom tensor
  registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
  CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
  (`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
  DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
  `--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
  `--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
  (`-rtr`). **Additive** — GGUF remains compatible.

**What mainline has that the forks lag on:**
- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
  architectures arrive late (PRs move fast, but the core is ~1.5y behind).
- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
  fork line, CUDA ecosystem (the fleet's 3090) is unreachable.

**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
even where GPU would be faster → lower prompt-processing speed. For a MoE that
keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.

## 4. Server-specific benefits

### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
  MoE**, which is ik_llama.cpp's core strength:
  - `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
    RAM→VRAM traffic dramatically vs shipping all experts.
  - `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
    MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
  Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
  K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
  `ngram-mod` drafts (MoEs want long drafts).
- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
  active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
  the GGUF before finalizing the offload recipe — both point at expert-aware
  offload being the right tool.
- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.

### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
  ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
  production `build-promptforge` `llama-server` already runs there.
- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
  validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
  draft-head speedup. Higher-risk change; only with a small soak-test first.
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
  IU4, TurboQuant, and MTEA.

## 6. KV cache quant table

Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
**KV-native** (designed for the cache) or **weight-family reused for KV** (a
model-weight quant the runtime also accepts in `-ctk`/`-ctv`).

**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
This is the true in-RAM cost per value, so it is what sets KV-cache size and
quantize/dequant throughput. A footnote marks the few types whose **nominal**
figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
number; the table uses the exact number throughout.

### 6a. All KV-usable types, by bit density (interleaved)

| bpw | Type | Origin | KV-native | Notes |
|-----|------|--------|-----------|-------|
| 32.0 | f32 | mainline * | yes | full precision |
| 16.0 | f16 | mainline * | yes | default |
| 16.0 | bf16 | mainline * | yes | |
| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
| 8.13 | **q8_0** | mainline * | yes | |
| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
| 6.63 | iq6_K | ik | weight→KV | IQK family |
| 6.56 | q6_K | mainline * | weight→KV | |
| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
| 5.50 | q5_K | mainline * | weight→KV | |
| 5.50 | iq5_K | ik | weight→KV | IQK family |
| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
| 4.50 | iq4_K | ik | weight→KV | IQK family |
| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
| 4.25 | mxfp4 | ik * | weight→KV | |
| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
| 3.44 | iq3_K | ik | weight→KV | IQK family |
| 3.44 | iq3_s | mainline * | weight→KV | |
| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
| 2.38 | iq2_K | ik | weight→KV | IQK family |
| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |

### 6b. Origin legend

- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
  `kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
  with `*` and `yes` in the KV-native column.
- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
  (add a third `V` type for values only). Marked `*` here.
- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
  ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
  `IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
  `mxfp4`. ik's `-ctk` accepts a broader set than mainline's.

### 6c. Trellis, IQK, MXFP4 — where they sit

- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
  ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
  quality per bit of the family; lowest-density entries in the whole table.
- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
  `iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
  `iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
  (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.

### 6d. Hadamard is a transform, not a row

`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
applied on top of a base KV quant**, not its own bit-density row. It spreads
quantization error, improving accuracy for the underlying type; ik's guidance is
**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
a packing transform on the K-family — and is likewise a modifier, not a quant.

### 6e. Server notes

- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
  `turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
  Strix is higher-risk (see §4).
- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
  FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
  confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
  — they are coupled (a quant change that drops FA off the path changes KV math).

## 5. Recommendations for our fleet

- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
  gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
  2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
  `offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
  much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
  + `ngram-mod`. Neither box needs ROCmFP4.
- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
  change (FlashAttention + KV-quant are coupled).

## See Also
- [[Llama Cpp Runtimes]] — positioning + quantization overview
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — quant vs hardware

## Citations
- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
  tools/quantize/README.md · serverAppConfigs-docs/speculative.md
- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, serverAppConfigs-docs/parameters.md, serverAppConfigs-docs/build.md) · DeepWiki · discussions#562
- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md