serverAppConfigs/docs/Research/Llama Cpp Runtimes.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	Llama.cpp Runtimes	Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo).	llama.cpp quantization ROCmFPX ik_llama inference amd nvidia	2026-09-01

Llama.cpp Runtimes

Comparison of three llama.cpp-derived inference runtimes used (or usable) on the two homelab boxes: mainline ggml-org/llama.cpp, charlie12345/ROCmFPX (what Dankclaw runs), and ik_llama.cpp (ikawrakow). The goal is to know, for each server, which quant formats are on the table and which runtime best fits the hardware.

Full code-level quantification is in Llama Cpp Code Diff. This page is the positioning + quantization summary.

Which runtime is which

Runtime	Repo	What it is	Primary backends	For our boxes
Mainline	ggml-org/llama.cpp	Canonical, general-purpose, one codebase for every backend	CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO	Baseline both boxes; the portable default
charlie12345/ROCmFPX	charlie12345/ROCmFPX (+ reference fork rocmfp4-llama, clean pkg rocmfp4)	Experimental AMD-focused GGUF weight quant family layered on llama.cpp	HIP/ROCm + Vulkan (CUDA disabled), CPU reference	Dankclaw (native Strix Halo)
ik_llama.cpp	ikawrakow/ik_llama.cpp	Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance	CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak)	Dankserver candidate (3090 + MoE)

Key lineage notes:

• Mainline moved from ggerganov/llama.cpp to the ggml-org org. Rolling commit hashes (the fleet pins b10121) plus newer vX.Y.Z tags. Pin a commit for reproducibility.
• ROCmFPX is an experimental feature family on the canonical main branch; three GitHub surfaces exist (reference fork, clean patch, pre-patched tree). It is AMD-only — CUDA is compiled out.
• ik_llama.cpp's core is an August-2024 llama.cpp snapshot maintained with a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all appeared here first, later partially merged upstream). The feature surface is current; the architecture baseline is ~1.5 years behind mainline.

Quantization summary

Mainline (what the fleet already ships)

• Base: FP32 / FP16 / BF16.
• K-quants (modern default, two-level block + double-quant): Q2_K … Q6_K, with _S/_M/_L precision mixes. Q4_K_M is the popular default; Q6_K is "almost lossless."
• i-quants (importance-matrix, non-linear, best quality/size at low bits but slower decode): IQ4_NL, IQ4_XS, IQ3_*, IQ2_*. Row-interleaved 3-bit family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
• Legacy: Q*_0/Q*_1 (fast, simple), ternary TQ1_0.
• KV cache quant: f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1; quantizing KV requires FlashAttention (-fa on).
• Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, ngram-mod).

ROCmFPX (Dankclaw's AMD weight formats)

• Integer GGUF weight formats only this fork reads: ROCmFP2/3/4/6/8 and ROCmI4 (W4A4 IU4 WMMA). These are custom ggml tensor types (ggml_type 100–108) with matching GGUF ftype (ROCmFP4 family 100–106; ROCmFPX family 110–113; ROCmI4 = 118). Stock llama.cpp, Ollama, and LM Studio cannot load them.
• Flagship ROCmFP4 (block_rocmfp4): 32 weights/block, two E4M3 scale bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10 units. Variants: STRIX / STRIX_LEAN, FAST, COHERENT, plus *_AGENT / *_STRIX_MTP draft-head presets.
• TurboQuant KV: Turbo3_0 (3.5 bpw) / Turbo4_0 (4.5 bpw) + TurboQuant V (f16 K) on the production model.
• ROCmI4 (W4A4): the gfx1151 native IU4 4-bit WMMA path (-DGGML_HIP_ROCMI4_W4A4=ON); lossy activation quant, exact int8 MMQ.

ik_llama.cpp (SOTA quants + CPU/hybrid)

• IQ* non-linear family; IQ4_NL gives lower perplexity than mainline Q4_K_M at the same size. Improved K-quants and the signature row-interleaved IQK / R4 / R8 packing (activates AVX-512/VNNI/NEON matmul paths that vanilla llama.cpp leaves cold).
• Trellis quants (IQ*_KT), MXFP4, Q8_KV fast KV type, first-class Bitnet.
• ⚠️ K-quant / -rtr trap (critical for the 3090): K-quants have no CUDA row-interleaved path; -rtr (run-time repack) forces CPU-resident K-quant tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and let --fit/-ot place experts.

Fit for our servers

Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA

Serves Ornith-1.5-35B-A3B CRACK, a 35B MoE (~36 experts; the CRACK GGUF README lists 256/8-active — verify n_expert from the GGUF).

• Mainline is the natural baseline: Q4_K_M/Q5_K_M, FlashAttention to shrink KV, GGML_CUDA_FORCE_MMQ=1 (default on RTX 30xx, lower VRAM), --cpu-moe to spill experts to the 62 GB RAM, and ngram-mod drafts (MoEs want long drafts).
• ik_llama.cpp is the more compelling upgrade here: fused MoE (-fmoe), offload-only-active-experts (-ooae) which — with only ~8 active experts — sharply cuts RAM→VRAM traffic, --fit, smart-expert-reduction (-ser), GDN fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care with the -rtr K-quant trap.
• ROCmFPX does not help a CUDA box (ROCm-only formats, CUDA disabled). The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4

Serves kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (the production build).

• ROCmFPX is the native fit: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV, 262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs Q4_K_M).
• ik_llama.cpp — ROCm is its weakest backend (gfx1151 not validated) and the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
• Mainline ROCm build works for correctness (standard quants) but loses FPX, IU4, TurboQuant, MTEA, and agent presets.

When to reach for which

• Stay mainline if you want one portable upstream for both boxes, the widest GGUF ecosystem, and standard FlashAttention/MTP. Best fit for Dankserver if you don't want the ik_llama.cpp build surface.
• Use ik_llama.cpp for Dankserver when the MoE offload and fused-MoE wins justify an extra build (watch -rtr + K-quants).
• Use ROCmFPX for Dankclaw — it is purpose-built for gfx1151.
• ROCmFPX is for the AMD fleet; CUDA/FastMTP for the NVIDIA box. Do not port ROCmFP4 to the 3090 via Vulkan as a primary path.

See Also

• Llama Cpp Code Diff — the code-level quantization diff + comparison table
• dealign-ornith-crack — the CRACK model on Dankserver
• kingjones-strix-mtp — the ROCmFP4 model on Dankclaw
• Quantization by Hardware Architecture — which quant fits which GPU/CPU
• SGLang — the structured-generation serving engine (the other lane)

Citations

• Mainline — https://github.com/ggml-org/llama.cpp
• charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX · rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 — https://github.com/charlie12345/rocmfp4
• ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs · https://ikawrakow-ik_llama-cpp.mintlify.app/
• ROCmFPX quant/enum/serve detail — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
• SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/