serverAppConfigs/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Llama.cpp Code Diff	Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards.	llama.cpp quantization ggml ROCmFPX ik_llama code-diff	2026-09-01

Llama.cpp Code Diff

Code-level diff of the three runtimes, focused on quantization code and the differences that affect our two servers. Positioning overview is in Llama Cpp Runtimes.

1. Quantization file map

Where quant lives in each runtime (paths relative to the repo root):

File / path	Mainline	ROCmFPX	ik_llama.cpp
ggml/include/ggml.h (enum ggml_type)	FP16/FP32/BF16, legacy Q*_0, K-quant types, IQ*_	adds Q4_0_ROCMFP4 … Q2_0_ROCMFPX, Q4_0_ROCMI4, Turbo3_0, Turbo4_0 (enums 100–108)	IQK non-linear (IQ4_NL etc.), Trellis IQ*_KT, MXFP4, Bitnet, Q8_KV
ggml/src/ggml-quants.*	K/I/legacy dequant, row-interleaved 3-bit	block_rocmfp4 unpack + E4M3 scaling, IU4 WMMA dispatch	iqk/* fused quant-gemms, IQK row-interleave, Trellis decode
tools/quantize/quantize.c	all mainline quants + --imatrix	ROCmFPX quantizer (STRIX/FAST/_AGENT presets)	IQK/Trellis/MXFP4 packers, --cache-type-k/v + Hadamard
gguf.cpp / gguf-io.c	standard ftype table	custom ftypes 100–106 (FP4), 110–113 (FPX), 118 (I4)	IQK/Trellis ftype tags
KV cache	f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1 (needs FlashAttention)	TurboQuant Turbo3_0/Turbo4_0/V	Q8_KV, IQK/Trellis/MXFP4 reused in -ctk, --k-cache-hadamard, --v-cache-hadamard

Two number spaces in ROCmFPX (authoritative ggml/include/ggml.h, sha c49ebdbd):

• ggml_type enum (tensor storage): Q4_0_ROCMFP4=100, Q4_0_ROCMFP4_FAST=101, Q6_0_ROCMFPX=102, Q8_0_ROCMFPX=103, Q3_0_ROCMFPX=104, Turbo3_0=105, Turbo4_0=106, Q2_0_ROCMFPX=107, Q4_0_ROCMI4=108, count 109.
• ggml_ftype (GGUF model-file): ROCmFP4 family 100–106 (STRIX=105, STRIX_LEAN=106), ROCmFPX family 110–113 (FP6=110, FP8=111, FP3=112, FP2=113), ROCmI4=118.

2. Comparison table

Runtime	Quant formats	Granularity	Custom ggml types	MoE	Attention accel	Hybrid GPU/CPU offload	Build backends
Mainline	K/I/legacy/TQ, imatrix, row-RI 3-bit	grouped/per-block + double-quant	none	native --cpu-moe	FlashAttention (multi-backend)	--n-gpu-layers, partial offload	CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN
charlie12345/ROCmFPX	ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV	block + E4M3 codebook	yes (100–108)	MTP draft head (bundled)	MTEA + TurboQuant FA (custom)	Strix-tuned MTP profile	HIP/ROCm + Vulkan (CUDA off), CPU ref
ik_llama.cpp	IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet	grouped + non-linear + row-interleaved	IQK/Trellis/MXFP4/Bitnet tags	fused MoE, -ooae, --fit, -ot, GDN	FlashMLA (DeepSeek MLA)	finest-grained override-tensor / cpu-moe / fit	CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak)

3. Code-difference narrative

What each fork ADDED that mainline lacks:

• ROCmFPX — ~10 new ggml_type entries, the block_rocmfp4 packed-nibble format, *_ROCMFPX/*_ROCMI4 ftypes, tensor-aware presets (*_AGENT, *_COHERENT, *_STRIX), block_rocmfp4 dequant + dispatch kernels, TurboQuant KV types, MTEA attention kernel. Mainline does not know these tensor types. It is a fork that adds a new backend: custom tensor registration + HIP/ROCm + Vulkan kernels + CPU reference. CUDA is disabled.
• ik_llama.cpp — new code under ggml/src/iqk/ (IQK GEMM + row-interleave), CUDA IQK flash-attn kernels (GGML_IQK_FA_ALL_QUANTS=ON), fused MoE (fnn_up+fnn_gate), fused delta-net (GDN) SSM ops, flash_fwd_mla for DeepSeek MLA/DSA, and hybrid-offload controls (-ot/--override-tensor, --cpu-moe, -ooae/--offload-only-active-experts, -ser/--smart-expert-reduction, --fit, -op/--offload-policy, split-mode graph). Run-time quant repack (-rtr). Additive — GGUF remains compatible.

What mainline has that the forks lag on:

• ik_llama.cpp: architecture baseline is Aug-2024 — newest architectures arrive late (PRs move fast, but the core is ~1.5y behind).
• ROCmFPX: AMD-first, experimental; narrower, must rebuild/re-quant per fork line, CUDA ecosystem (the fleet's 3090) is unreachable.

The one caveat that changes a build (ik_llama.cpp + -rtr): K-quants have no CUDA row-interleaved implementation. With -rtr, every CPU-resident K-quant tensor is repacked to row-interleaved and then run on CPU even where GPU would be faster → lower prompt-processing speed. For a MoE that keeps experts on CPU (--cpu-moe), this is a real hit. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and place tensors with --fit/-ot.

4. Server-specific benefits

Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)

• The winning lever for a low-VRAM CUDA MoE is expert-aware offload + fused MoE, which is ik_llama.cpp's core strength:
  • -ooae/--offload-only-active-experts — with ~8 active experts this cuts RAM→VRAM traffic dramatically vs shipping all experts.
  • -fmoe/--fused-moe, GDN fusion (the CRACK model is a hybrid GatedDeltaNet + MoE), --fit, -ot/--override-tensor, --cpu-moe.
• Quant pick for the 3090: Q4_K_M/Q5_K_M on mainline, or an IQK/R4 / Trellis / MXFP4 variant on ik_llama.cpp that has a CUDA path. Avoid raw K-quant + -rtr. Enable FlashAttention; GGML_CUDA_FORCE_MMQ=1; ngram-mod drafts (MoEs want long drafts).
• ⚠️ Expert-count discrepancy: the CRACK GGUF README says 256 experts / 8 active; the fleet brief cites ~36. Verify n_expert / n_expert_used from the GGUF before finalizing the offload recipe — both point at expert-aware offload being the right tool.
• Mainline only? Fine: --cpu-moe + Q4_K_M + FlashAttention + ngram-mod.
• ROCmFPX? No — ROCm-only formats, CUDA disabled on the builds.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP

• ROCmFPX is the natural fit and stays the choice: native IU4 (ROCmI4), ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The production build-promptforge llama-server already runs there.
• ik_llama.cpp on Strix? ROCm is its weakest backend (gfx1151 not validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP draft-head speedup. Higher-risk change; only with a small soak-test first.
• Mainline on Strix works for correctness with standard quants but loses FPX, IU4, TurboQuant, and MTEA.

6. KV cache quant table

Bit-density-ordered comparison of everything usable in the KV cache (-ctk key / -ctv value, and draft -ctkd/-ctvd). Order is high → low bits-per- value. Columns mark origin (* = mainline-provided) and whether the type is KV-native (designed for the cache) or weight-family reused for KV (a model-weight quant the runtime also accepts in -ctk/-ctv).

How bpw is read here. bpw = ggml_type_size × 8 / QK, computed from the exact block structs in ggml.h / ggml-common.h (QK_K = 256, fp16 scale). This is the true in-RAM cost per value, so it is what sets KV-cache size and quantize/dequant throughput. A footnote marks the few types whose nominal figure (older 1-byte-scale layout / data-bits-only) diverges from this exact number; the table uses the exact number throughout.

6a. All KV-usable types, by bit density (interleaved)

bpw	Type	Origin	KV-native	Notes
32.0	f32	mainline *	yes	full precision
16.0	f16	mainline *	yes	default
16.0	bf16	mainline *	yes
9.25	q8_K	mainline *	weight	rare in KV; per-16-block sums; ik Q8_KV is the fast KV version
8.13	q8_0	mainline *	yes
8.06	q8_KV	ik *	yes	the dedicated fast KV cache type (-ctk q8_KV); Q8_0-shaped, 1 shared scale
6.63	iq6_K	ik	weight→KV	IQK family
6.56	q6_K	mainline *	weight→KV
6.00	q5_1	mainline *	yes	(nominal 5.06)
5.50	q5_0	mainline *	yes	(nominal 5.06)
5.50	q5_K	mainline *	weight→KV
5.50	iq5_K	ik	weight→KV	IQK family
4.50	q4_0	mainline *	yes	(nominal 4.25)
4.50	q4_1	mainline *	yes	(nominal 4.25)
4.50	iq4_nl	mainline *	yes	(nominal 4.0)
4.50	iq4_K	ik	weight→KV	IQK family
4.5	Turbo4_0	ROCmFPX *	yes	TurboQuant; ~4.5
4.25	iq4_ks	ik	weight→KV	IQK subsample
4.25	iq4_xs	mainline *	weight→KV	popular weight format, also usable in KV
4.25	mxfp4	ik *	weight→KV
3.5	Turbo3_0	ROCmFPX *	yes	TurboQuant; ~3.5
3.44	iq3_K	ik	weight→KV	IQK family
3.44	iq3_s	mainline *	weight→KV
2.69	iq2_kl	ik	weight→KV	IQK subsample
2.38	iq2_K	ik	weight→KV	IQK family
2.19	iq2_ks	ik	weight→KV	IQK subsample
2.13	iq2_kt	ik *	weight→KV	Trellis (lattice)
1.75	iq1_kt	ik *	weight→KV	Trellis (lattice)

6b. Origin legend

• Mainline KV-native (-ctk/-ctv = exactly these 9, from common/arg.cpp kv_cache_types): f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. Marked with * and yes in the KV-native column.
• ROCmFPX KV-native: turbo3_0, turbo4_0 — TurboQuant's dedicated KV formats (add a third V type for values only). Marked * here.
• ik additions: q8_KV (fast KV, dedicated), plus the weight-family quants ik lets you reuse in -ctk/-ctv — the IQK family (iq*_K, plus the IKS/IKL subsamples) and the Trellis lattice family (iq*_KT), and mxfp4. ik's -ctk accepts a broader set than mainline's.

6c. Trellis, IQK, MXFP4 — where they sit

• Trellis (IQ*_KT) — ik's lattice-code quants, mirror the IQ1..IQ4 bit ladder: iq1_kt 1.75 → iq2_kt 2.13 → iq3_kt 3.13 → iq4_kt 4.00. Highest quality per bit of the family; lowest-density entries in the whole table.
• IQK (IQ*_K) — ik's improved K-quants: iq2_k 2.38, iq3_k 3.44, iq4_k 4.50, iq5_k 5.50, iq6_k 6.63, plus subsamples iq2_kl 2.69 / iq2_ks 2.19 / iq4_ks 4.25. They sit alongside the mainline K-family (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
• MXFP4 (mxfp4, 4.25) — the mixed-precision FP4 cache quant; ik's entry.

6d. Hadamard is a transform, not a row

--k-cache-hadamard / --v-cache-hadamard (ik) is a structured transform applied on top of a base KV quant, not its own bit-density row. It spreads quantization error, improving accuracy for the underlying type; ik's guidance is best below Q6_0. Row-interleaving (the R4/R8 variants) is the same idea — a packing transform on the K-family — and is likewise a modifier, not a quant.

6e. Server notes

• Dankclaw (production, ROCmFP4 STRIX MTP): KV stays on turbo4_0 / turbo3_0 (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on Strix is higher-risk (see §4).
• Dankserver (3090 MoE): -ctk q8_KV or q4_K/q5_K (mainline) with FlashAttention + KV-quant; on ik, iq4_k/iq4_kt/iq2_k if a CUDA path is confirmed for that type. Keep FlashAttention and the KV quant version in lockstep — they are coupled (a quant change that drops FA off the path changes KV math).

5. Recommendations for our fleet

• Dankclaw: keep charlie12345/ROCmFPX (already production-tuned for gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (Q2_0_ROCMFPX, 2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
• Dankserver: try ik_llama.cpp (CUDA-90 + AVX-512 build) for fused-MoE + offload-only-active-experts, staying off raw K-quant -rtr. If that is too much surface, stay on mainline with --cpu-moe + Q4_K_M + FlashAttention
  • ngram-mod. Neither box needs ROCmFP4.
• Common: pin a commit per fleet line; re-test after any quant/KV/context change (FlashAttention + KV-quant are coupled).

See Also

• Llama Cpp Runtimes — positioning + quantization overview
• dealign-ornith-crack — the CRACK MoE on Dankserver
• frank-dragoon — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
• Quantization by Hardware Architecture — quant vs hardware

Citations

• Mainline quant + backends — https://github.com/ggml-org/llama.cpp · tools/quantize/README.md · serverAppConfigs-docs/speculative.md
• ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
• ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, serverAppConfigs-docs/parameters.md, serverAppConfigs-docs/build.md) · DeepWiki · discussions#562
• ik_llama.cpp -rtr K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md