dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
docs/retire-cold-voice-ladder
serverAppConfigs/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md
T
Frank 5829a0bfe8 docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention 
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
2026-08-28 02:20:06 -07:00

13 KiB
Raw Permalink Blame History

type, title, description, tags, timestamp
type title description tags timestamp
research Llama.cpp Code Diff Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
llama.cpp
quantization
ggml
ROCmFPX
ik_llama
code-diff
2026-09-01

Llama.cpp Code Diff

Code-level diff of the three runtimes, focused on quantization code and the differences that affect our two servers. Positioning overview is in Llama Cpp Runtimes.

1. Quantization file map

Where quant lives in each runtime (paths relative to the repo root):

File / path Mainline ROCmFPX ik_llama.cpp
ggml/include/ggml.h (enum ggml_type) FP16/FP32/BF16, legacy Q*_0, K-quant types, IQ*_ adds Q4_0_ROCMFP4 … Q2_0_ROCMFPX, Q4_0_ROCMI4, Turbo3_0, Turbo4_0 (enums 100–108) IQK non-linear (IQ4_NL etc.), Trellis IQ*_KT, MXFP4, Bitnet, Q8_KV
ggml/src/ggml-quants.* K/I/legacy dequant, row-interleaved 3-bit block_rocmfp4 unpack + E4M3 scaling, IU4 WMMA dispatch iqk/* fused quant-gemms, IQK row-interleave, Trellis decode
tools/quantize/quantize.c all mainline quants + --imatrix ROCmFPX quantizer (STRIX/FAST/_AGENT presets) IQK/Trellis/MXFP4 packers, --cache-type-k/v + Hadamard
gguf.cpp / gguf-io.c standard ftype table custom ftypes 100–106 (FP4), 110–113 (FPX), 118 (I4) IQK/Trellis ftype tags
KV cache f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1 (needs FlashAttention) TurboQuant Turbo3_0/Turbo4_0/V Q8_KV, IQK/Trellis/MXFP4 reused in -ctk, --k-cache-hadamard, --v-cache-hadamard

Two number spaces in ROCmFPX (authoritative ggml/include/ggml.h, sha c49ebdbd):

  • ggml_type enum (tensor storage): Q4_0_ROCMFP4=100, Q4_0_ROCMFP4_FAST=101, Q6_0_ROCMFPX=102, Q8_0_ROCMFPX=103, Q3_0_ROCMFPX=104, Turbo3_0=105, Turbo4_0=106, Q2_0_ROCMFPX=107, Q4_0_ROCMI4=108, count 109.
  • ggml_ftype (GGUF model-file): ROCmFP4 family 100–106 (STRIX=105, STRIX_LEAN=106), ROCmFPX family 110–113 (FP6=110, FP8=111, FP3=112, FP2=113), ROCmI4=118.

2. Comparison table

Runtime Quant formats Granularity Custom ggml types MoE Attention accel Hybrid GPU/CPU offload Build backends
Mainline K/I/legacy/TQ, imatrix, row-RI 3-bit grouped/per-block + double-quant none native --cpu-moe FlashAttention (multi-backend) --n-gpu-layers, partial offload CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN
charlie12345/ROCmFPX ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV block + E4M3 codebook yes (100–108) MTP draft head (bundled) MTEA + TurboQuant FA (custom) Strix-tuned MTP profile HIP/ROCm + Vulkan (CUDA off), CPU ref
ik_llama.cpp IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet grouped + non-linear + row-interleaved IQK/Trellis/MXFP4/Bitnet tags fused MoE, -ooae, --fit, -ot, GDN FlashMLA (DeepSeek MLA) finest-grained override-tensor / cpu-moe / fit CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak)

3. Code-difference narrative

What each fork ADDED that mainline lacks:

  • ROCmFPX — ~10 new ggml_type entries, the block_rocmfp4 packed-nibble format, *_ROCMFPX/*_ROCMI4 ftypes, tensor-aware presets (*_AGENT, *_COHERENT, *_STRIX), block_rocmfp4 dequant + dispatch kernels, TurboQuant KV types, MTEA attention kernel. Mainline does not know these tensor types. It is a fork that adds a new backend: custom tensor registration + HIP/ROCm + Vulkan kernels + CPU reference. CUDA is disabled.
  • ik_llama.cpp — new code under ggml/src/iqk/ (IQK GEMM + row-interleave), CUDA IQK flash-attn kernels (GGML_IQK_FA_ALL_QUANTS=ON), fused MoE (fnn_up+fnn_gate), fused delta-net (GDN) SSM ops, flash_fwd_mla for DeepSeek MLA/DSA, and hybrid-offload controls (-ot/--override-tensor, --cpu-moe, -ooae/--offload-only-active-experts, -ser/--smart-expert-reduction, --fit, -op/--offload-policy, split-mode graph). Run-time quant repack (-rtr). Additive — GGUF remains compatible.

What mainline has that the forks lag on:

  • ik_llama.cpp: architecture baseline is Aug-2024 — newest architectures arrive late (PRs move fast, but the core is ~1.5y behind).
  • ROCmFPX: AMD-first, experimental; narrower, must rebuild/re-quant per fork line, CUDA ecosystem (the fleet's 3090) is unreachable.

The one caveat that changes a build (ik_llama.cpp + -rtr): K-quants have no CUDA row-interleaved implementation. With -rtr, every CPU-resident K-quant tensor is repacked to row-interleaved and then run on CPU even where GPU would be faster → lower prompt-processing speed. For a MoE that keeps experts on CPU (--cpu-moe), this is a real hit. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and place tensors with --fit/-ot.

4. Server-specific benefits

Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)

  • The winning lever for a low-VRAM CUDA MoE is expert-aware offload + fused MoE, which is ik_llama.cpp's core strength:
    • -ooae/--offload-only-active-experts — with ~8 active experts this cuts RAM→VRAM traffic dramatically vs shipping all experts.
    • -fmoe/--fused-moe, GDN fusion (the CRACK model is a hybrid GatedDeltaNet + MoE), --fit, -ot/--override-tensor, --cpu-moe.
  • Quant pick for the 3090: Q4_K_M/Q5_K_M on mainline, or an IQK/R4 / Trellis / MXFP4 variant on ik_llama.cpp that has a CUDA path. Avoid raw K-quant + -rtr. Enable FlashAttention; GGML_CUDA_FORCE_MMQ=1; ngram-mod drafts (MoEs want long drafts).
  • ⚠️ Expert-count discrepancy: the CRACK GGUF README says 256 experts / 8 active; the fleet brief cites ~36. Verify n_expert / n_expert_used from the GGUF before finalizing the offload recipe — both point at expert-aware offload being the right tool.
  • Mainline only? Fine: --cpu-moe + Q4_K_M + FlashAttention + ngram-mod.
  • ROCmFPX? No — ROCm-only formats, CUDA disabled on the builds.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP

  • ROCmFPX is the natural fit and stays the choice: native IU4 (ROCmI4), ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The production build-promptforge llama-server already runs there.
  • ik_llama.cpp on Strix? ROCm is its weakest backend (gfx1151 not validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP draft-head speedup. Higher-risk change; only with a small soak-test first.
  • Mainline on Strix works for correctness with standard quants but loses FPX, IU4, TurboQuant, and MTEA.

6. KV cache quant table

Bit-density-ordered comparison of everything usable in the KV cache (-ctk key / -ctv value, and draft -ctkd/-ctvd). Order is high → low bits-per- value. Columns mark origin (* = mainline-provided) and whether the type is KV-native (designed for the cache) or weight-family reused for KV (a model-weight quant the runtime also accepts in -ctk/-ctv).

How bpw is read here. bpw = ggml_type_size × 8 / QK, computed from the exact block structs in ggml.h / ggml-common.h (QK_K = 256, fp16 scale). This is the true in-RAM cost per value, so it is what sets KV-cache size and quantize/dequant throughput. A footnote marks the few types whose nominal figure (older 1-byte-scale layout / data-bits-only) diverges from this exact number; the table uses the exact number throughout.

6a. All KV-usable types, by bit density (interleaved)

bpw Type Origin KV-native Notes
32.0 f32 mainline * yes full precision
16.0 f16 mainline * yes default
16.0 bf16 mainline * yes
9.25 q8_K mainline * weight rare in KV; per-16-block sums; ik Q8_KV is the fast KV version
8.13 q8_0 mainline * yes
8.06 q8_KV ik * yes the dedicated fast KV cache type (-ctk q8_KV); Q8_0-shaped, 1 shared scale
6.63 iq6_K ik weight→KV IQK family
6.56 q6_K mainline * weight→KV
6.00 q5_1 mainline * yes (nominal 5.06)
5.50 q5_0 mainline * yes (nominal 5.06)
5.50 q5_K mainline * weight→KV
5.50 iq5_K ik weight→KV IQK family
4.50 q4_0 mainline * yes (nominal 4.25)
4.50 q4_1 mainline * yes (nominal 4.25)
4.50 iq4_nl mainline * yes (nominal 4.0)
4.50 iq4_K ik weight→KV IQK family
4.5 Turbo4_0 ROCmFPX * yes TurboQuant; ~4.5
4.25 iq4_ks ik weight→KV IQK subsample
4.25 iq4_xs mainline * weight→KV popular weight format, also usable in KV
4.25 mxfp4 ik * weight→KV
3.5 Turbo3_0 ROCmFPX * yes TurboQuant; ~3.5
3.44 iq3_K ik weight→KV IQK family
3.44 iq3_s mainline * weight→KV
2.69 iq2_kl ik weight→KV IQK subsample
2.38 iq2_K ik weight→KV IQK family
2.19 iq2_ks ik weight→KV IQK subsample
2.13 iq2_kt ik * weight→KV Trellis (lattice)
1.75 iq1_kt ik * weight→KV Trellis (lattice)

6b. Origin legend

  • Mainline KV-native (-ctk/-ctv = exactly these 9, from common/arg.cpp kv_cache_types): f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1. Marked with * and yes in the KV-native column.
  • ROCmFPX KV-native: turbo3_0, turbo4_0 — TurboQuant's dedicated KV formats (add a third V type for values only). Marked * here.
  • ik additions: q8_KV (fast KV, dedicated), plus the weight-family quants ik lets you reuse in -ctk/-ctv — the IQK family (iq*_K, plus the IKS/IKL subsamples) and the Trellis lattice family (iq*_KT), and mxfp4. ik's -ctk accepts a broader set than mainline's.

6c. Trellis, IQK, MXFP4 — where they sit

  • Trellis (IQ*_KT) — ik's lattice-code quants, mirror the IQ1..IQ4 bit ladder: iq1_kt 1.75 → iq2_kt 2.13 → iq3_kt 3.13 → iq4_kt 4.00. Highest quality per bit of the family; lowest-density entries in the whole table.
  • IQK (IQ*_K) — ik's improved K-quants: iq2_k 2.38, iq3_k 3.44, iq4_k 4.50, iq5_k 5.50, iq6_k 6.63, plus subsamples iq2_kl 2.69 / iq2_ks 2.19 / iq4_ks 4.25. They sit alongside the mainline K-family (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
  • MXFP4 (mxfp4, 4.25) — the mixed-precision FP4 cache quant; ik's entry.

6d. Hadamard is a transform, not a row

--k-cache-hadamard / --v-cache-hadamard (ik) is a structured transform applied on top of a base KV quant, not its own bit-density row. It spreads quantization error, improving accuracy for the underlying type; ik's guidance is best below Q6_0. Row-interleaving (the R4/R8 variants) is the same idea — a packing transform on the K-family — and is likewise a modifier, not a quant.

6e. Server notes

  • Dankclaw (production, ROCmFP4 STRIX MTP): KV stays on turbo4_0 / turbo3_0 (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on Strix is higher-risk (see §4).
  • Dankserver (3090 MoE): -ctk q8_KV or q4_K/q5_K (mainline) with FlashAttention + KV-quant; on ik, iq4_k/iq4_kt/iq2_k if a CUDA path is confirmed for that type. Keep FlashAttention and the KV quant version in lockstep — they are coupled (a quant change that drops FA off the path changes KV math).

5. Recommendations for our fleet

  • Dankclaw: keep charlie12345/ROCmFPX (already production-tuned for gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (Q2_0_ROCMFPX, 2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
  • Dankserver: try ik_llama.cpp (CUDA-90 + AVX-512 build) for fused-MoE + offload-only-active-experts, staying off raw K-quant -rtr. If that is too much surface, stay on mainline with --cpu-moe + Q4_K_M + FlashAttention
    • ngram-mod. Neither box needs ROCmFP4.
  • Common: pin a commit per fleet line; re-test after any quant/KV/context change (FlashAttention + KV-quant are coupled).

See Also

Citations

Reference in New Issue View Git Blame Copy Permalink