serverAppConfigs/docs/Research/Llama Cpp Runtimes.md

---
type: research
title: Llama.cpp Runtimes
description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
tags:
  - llama.cpp
  - quantization
  - ROCmFPX
  - ik_llama
  - inference
  - amd
  - nvidia
timestamp: 2026-09-01
---

# Llama.cpp Runtimes

Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
(ikawrakow). The goal is to know, for each server, which quant formats are on
the table and which runtime best fits the hardware.

Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
positioning + quantization summary.

## Which runtime is which

| Runtime | Repo | What it is | Primary backends | For our boxes |
|---------|------|-----------|------------------|---------------|
| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |

Key lineage notes:
- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
  commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
  commit for reproducibility.
- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
  three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
  It is **AMD-only** — CUDA is compiled out.
- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
  a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
  appeared here first, later partially merged upstream). The feature surface is
  current; the architecture baseline is ~1.5 years behind mainline.

## Quantization summary

### Mainline (what the fleet already ships)
- **Base:** FP32 / FP16 / BF16.
- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
  `Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
  `Q6_K` is "almost lossless."
- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
  slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
  family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
  requires FlashAttention (`-fa on`).
- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).

### ROCmFPX (Dankclaw's AMD weight formats)
- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
  **ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
  (**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
  `100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
  Ollama, and LM Studio **cannot load them**.
- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
  bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
  units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
  / `*_STRIX_MTP` draft-head presets.
- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
  (f16 K) on the production model.
- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
  lossy activation quant, exact int8 MMQ.

### ik_llama.cpp (SOTA quants + CPU/hybrid)
- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
  `Q4_K_M` at the same size. Improved **K-quants** and the signature
  **row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
  matmul paths that vanilla llama.cpp leaves cold).
- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
  **Bitnet**.
- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
  row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
  tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
  MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.

## Fit for our servers

### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
  shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
  `--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
  want long drafts).
- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
  **offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
  sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
  fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
  with the `-rtr` K-quant trap.
- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
  The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.

### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
  262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
  the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
  IU4, TurboQuant, MTEA, and agent presets.

## When to reach for which
- **Stay mainline** if you want one portable upstream for both boxes, the widest
  GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
  if you don't want the ik_llama.cpp build surface.
- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
  justify an extra build (watch `-rtr` + K-quants).
- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
  port ROCmFP4 to the 3090 via Vulkan as a primary path.

## See Also
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
- [[SGLang]] — the structured-generation serving engine (the other lane)

## Citations
- Mainline — https://github.com/ggml-org/llama.cpp
- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
  rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
  https://github.com/charlie12345/rocmfp4
- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
  https://ikawrakow-ik_llama-cpp.mintlify.app/
- ROCmFPX quant/enum/serve detail —
  https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/