serverAppConfigs/docs/Research/Llama Cpp Code Diff.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
research	Llama.cpp Code Diff	Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).	llama.cpp quantization ggml ROCmFPX ik_llama code-diff	2026-09-01

Llama.cpp Code Diff

Code-level diff of the three runtimes, focused on quantization code and the differences that affect our two servers. Positioning overview is in Llama Cpp Runtimes.

1. Quantization file map

Where quant lives in each runtime (paths relative to the repo root):

File / path	Mainline	ROCmFPX	ik_llama.cpp
ggml/include/ggml.h (enum ggml_type)	FP16/FP32/BF16, legacy Q*_0, K-quant types, IQ*_	adds Q4_0_ROCMFP4 … Q2_0_ROCMFPX, Q4_0_ROCMI4, Turbo3_0, Turbo4_0 (enums 100–108)	IQK non-linear (IQ4_NL etc.), Trellis IQ*_KT, MXFP4, Bitnet, Q8_KV
ggml/src/ggml-quants.*	K/I/legacy dequant, row-interleaved 3-bit	block_rocmfp4 unpack + E4M3 scaling, IU4 WMMA dispatch	iqk/* fused quant-gemms, IQK row-interleave, Trellis decode
tools/quantize/quantize.c	all mainline quants + --imatrix	ROCmFPX quantizer (STRIX/FAST/_AGENT presets)	IQK/Trellis/MXFP4 packers, --cache-type-k/v + Hadamard
gguf.cpp / gguf-io.c	standard ftype table	custom ftypes 100–106 (FP4), 110–113 (FPX), 118 (I4)	IQK/Trellis ftype tags
KV cache	f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1 (needs FlashAttention)	TurboQuant Turbo3_0/Turbo4_0/V	Q8_KV, --k-cache-hadamard, --v-cache-hadamard

Two number spaces in ROCmFPX (authoritative ggml/include/ggml.h, sha c49ebdbd):

• ggml_type enum (tensor storage): Q4_0_ROCMFP4=100, Q4_0_ROCMFP4_FAST=101, Q6_0_ROCMFPX=102, Q8_0_ROCMFPX=103, Q3_0_ROCMFPX=104, Turbo3_0=105, Turbo4_0=106, Q2_0_ROCMFPX=107, Q4_0_ROCMI4=108, count 109.
• ggml_ftype (GGUF model-file): ROCmFP4 family 100–106 (STRIX=105, STRIX_LEAN=106), ROCmFPX family 110–113 (FP6=110, FP8=111, FP3=112, FP2=113), ROCmI4=118.

2. Comparison table

Runtime	Quant formats	Granularity	Custom ggml types	MoE	Attention accel	Hybrid GPU/CPU offload	Build backends
Mainline	K/I/legacy/TQ, imatrix, row-RI 3-bit	grouped/per-block + double-quant	none	native --cpu-moe	FlashAttention (multi-backend)	--n-gpu-layers, partial offload	CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN
charlie12345/ROCmFPX	ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV	block + E4M3 codebook	yes (100–108)	MTP draft head (bundled)	MTEA + TurboQuant FA (custom)	Strix-tuned MTP profile	HIP/ROCm + Vulkan (CUDA off), CPU ref
ik_llama.cpp	IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet	grouped + non-linear + row-interleaved	IQK/Trellis/MXFP4/Bitnet tags	fused MoE, -ooae, --fit, -ot, GDN	FlashMLA (DeepSeek MLA)	finest-grained override-tensor / cpu-moe / fit	CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak)

3. Code-difference narrative

What each fork ADDED that mainline lacks:

• ROCmFPX — ~10 new ggml_type entries, the block_rocmfp4 packed-nibble format, *_ROCMFPX/*_ROCMI4 ftypes, tensor-aware presets (*_AGENT, *_COHERENT, *_STRIX), block_rocmfp4 dequant + dispatch kernels, TurboQuant KV types, MTEA attention kernel. Mainline does not know these tensor types. It is a fork that adds a new backend: custom tensor registration + HIP/ROCm + Vulkan kernels + CPU reference. CUDA is disabled.
• ik_llama.cpp — new code under ggml/src/iqk/ (IQK GEMM + row-interleave), CUDA IQK flash-attn kernels (GGML_IQK_FA_ALL_QUANTS=ON), fused MoE (fnn_up+fnn_gate), fused delta-net (GDN) SSM ops, flash_fwd_mla for DeepSeek MLA/DSA, and hybrid-offload controls (-ot/--override-tensor, --cpu-moe, -ooae/--offload-only-active-experts, -ser/--smart-expert-reduction, --fit, -op/--offload-policy, split-mode graph). Run-time quant repack (-rtr). Additive — GGUF remains compatible.

What mainline has that the forks lag on:

• ik_llama.cpp: architecture baseline is Aug-2024 — newest architectures arrive late (PRs move fast, but the core is ~1.5y behind).
• ROCmFPX: AMD-first, experimental; narrower, must rebuild/re-quant per fork line, CUDA ecosystem (the fleet's 3090) is unreachable.

The one caveat that changes a build (ik_llama.cpp + -rtr): K-quants have no CUDA row-interleaved implementation. With -rtr, every CPU-resident K-quant tensor is repacked to row-interleaved and then run on CPU even where GPU would be faster → lower prompt-processing speed. For a MoE that keeps experts on CPU (--cpu-moe), this is a real hit. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and place tensors with --fit/-ot.

4. Server-specific benefits

Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)

• The winning lever for a low-VRAM CUDA MoE is expert-aware offload + fused MoE, which is ik_llama.cpp's core strength:
  • -ooae/--offload-only-active-experts — with ~8 active experts this cuts RAM→VRAM traffic dramatically vs shipping all experts.
  • -fmoe/--fused-moe, GDN fusion (the CRACK model is a hybrid GatedDeltaNet + MoE), --fit, -ot/--override-tensor, --cpu-moe.
• Quant pick for the 3090: Q4_K_M/Q5_K_M on mainline, or an IQK/R4 / Trellis / MXFP4 variant on ik_llama.cpp that has a CUDA path. Avoid raw K-quant + -rtr. Enable FlashAttention; GGML_CUDA_FORCE_MMQ=1; ngram-mod drafts (MoEs want long drafts).
• ⚠️ Expert-count discrepancy: the CRACK GGUF README says 256 experts / 8 active; the fleet brief cites ~36. Verify n_expert / n_expert_used from the GGUF before finalizing the offload recipe — both point at expert-aware offload being the right tool.
• Mainline only? Fine: --cpu-moe + Q4_K_M + FlashAttention + ngram-mod.
• ROCmFPX? No — ROCm-only formats, CUDA disabled on the builds.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP

• ROCmFPX is the natural fit and stays the choice: native IU4 (ROCmI4), ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The production build-promptforge llama-server already runs there.
• ik_llama.cpp on Strix? ROCm is its weakest backend (gfx1151 not validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP draft-head speedup. Higher-risk change; only with a small soak-test first.
• Mainline on Strix works for correctness with standard quants but loses FPX, IU4, TurboQuant, and MTEA.

5. Recommendations for our fleet

• Dankclaw: keep charlie12345/ROCmFPX (already production-tuned for gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (Q2_0_ROCMFPX, 2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
• Dankserver: try ik_llama.cpp (CUDA-90 + AVX-512 build) for fused-MoE + offload-only-active-experts, staying off raw K-quant -rtr. If that is too much surface, stay on mainline with --cpu-moe + Q4_K_M + FlashAttention
  • ngram-mod. Neither box needs ROCmFP4.
• Common: pin a commit per fleet line; re-test after any quant/KV/context change (FlashAttention + KV-quant are coupled).

See Also

• Llama Cpp Runtimes — positioning + quantization overview
• dealign-ornith-crack — the CRACK MoE on Dankserver
• kingjones-strix-mtp — the ROCmFP4 model on Dankclaw
• Quantization by Hardware Architecture — quant vs hardware

Citations

• Mainline quant + backends — https://github.com/ggml-org/llama.cpp · tools/quantize/README.md · docs/speculative.md
• ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
• ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, docs/parameters.md, docs/build.md) · DeepWiki · discussions#562
• ik_llama.cpp -rtr K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md