---
type: research
title: Llama.cpp Code Diff
description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
tags:
  - llama.cpp
  - quantization
  - ggml
  - ROCmFPX
  - ik_llama
  - code-diff
timestamp: 2026-09-01
---

# Llama.cpp Code Diff

Code-level diff of the three runtimes, focused on **quantization code** and the
differences that affect our two servers. Positioning overview is in
[[Llama Cpp Runtimes]].

## 1. Quantization file map

Where quant lives in each runtime (paths relative to the repo root):

| File / path | Mainline | ROCmFPX | ik_llama.cpp |
|-------------|----------|---------|--------------|
| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` |

Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
`c49ebdbd`):
- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
  `Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
  `Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
  `STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
  FP2=113), `ROCmI4`=118.

## 2. Comparison table

| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |

## 3. Code-difference narrative

**What each fork ADDED that mainline lacks:**
- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
  format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
  `*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
  TurboQuant KV types, MTEA attention kernel. Mainline does not know these
  tensor types. It is a *fork* that adds a new backend: custom tensor
  registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
  CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
  (`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
  DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
  `--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
  `--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
  (`-rtr`). **Additive** — GGUF remains compatible.

**What mainline has that the forks lag on:**
- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
  architectures arrive late (PRs move fast, but the core is ~1.5y behind).
- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
  fork line, CUDA ecosystem (the fleet's 3090) is unreachable.

**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
even where GPU would be faster → lower prompt-processing speed. For a MoE that
keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.

## 4. Server-specific benefits

### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
  MoE**, which is ik_llama.cpp's core strength:
  - `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
    RAM→VRAM traffic dramatically vs shipping all experts.
  - `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
    MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
  Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
  K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
  `ngram-mod` drafts (MoEs want long drafts).
- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
  active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
  the GGUF before finalizing the offload recipe — both point at expert-aware
  offload being the right tool.
- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.

### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
  ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
  production `build-promptforge` `llama-server` already runs there.
- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
  validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
  draft-head speedup. Higher-risk change; only with a small soak-test first.
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
  IU4, TurboQuant, and MTEA.

## 5. Recommendations for our fleet

- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
  gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
  2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
  `offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
  much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
  + `ngram-mod`. Neither box needs ROCmFP4.
- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
  change (FlashAttention + KV-quant are coupled).

## See Also
- [[Llama Cpp Runtimes]] — positioning + quantization overview
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
- [[Quantization by Hardware Architecture]] — quant vs hardware

## Citations
- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
  tools/quantize/README.md · docs/speculative.md
- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, docs/parameters.md, docs/build.md) · DeepWiki · discussions#562
- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md