Frank
929796352a
Fan out subagents to compare mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp, then add their findings to the project knowledge base: - docs/Research/Llama Cpp Runtimes.md (positioning + quantization + server fit) - docs/Research/Llama Cpp Code Diff.md (code-level quant diff, table, recommendations) - docs/Research/SGLang.md (what SGLang is, comparison, our-boxes fit) - docs/Research/index.md (Research progressive-disclosure index) Cross-linked from docs/index.md (root) and docs/LLM/index.md. AGENTS.md: - Add Documentation Authoring section: docs/ is an OKF bundle; point doc authors at the canonical spec in the system-config knowledge base (~/.pi/agent/docs/references/okf-spec.md), not the orphaned docs/references/okf-spec.md; consolidate read-index/one-concept-per-file/ progressive-disclosure/citation rules. - Map AGENTS.md to docs_completeness.bats in the focused pre-commit hook (it was last committed before the hook existed and had no mapping).
134 lines
8.4 KiB
Markdown
134 lines
8.4 KiB
Markdown
---
|
||
type: research
|
||
title: Llama.cpp Code Diff
|
||
description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||
tags:
|
||
- llama.cpp
|
||
- quantization
|
||
- ggml
|
||
- ROCmFPX
|
||
- ik_llama
|
||
- code-diff
|
||
timestamp: 2026-09-01
|
||
---
|
||
|
||
# Llama.cpp Code Diff
|
||
|
||
Code-level diff of the three runtimes, focused on **quantization code** and the
|
||
differences that affect our two servers. Positioning overview is in
|
||
[[Llama Cpp Runtimes]].
|
||
|
||
## 1. Quantization file map
|
||
|
||
Where quant lives in each runtime (paths relative to the repo root):
|
||
|
||
| File / path | Mainline | ROCmFPX | ik_llama.cpp |
|
||
|-------------|----------|---------|--------------|
|
||
| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
|
||
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
|
||
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
|
||
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
|
||
| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` |
|
||
|
||
Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
|
||
`c49ebdbd`):
|
||
- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
|
||
`Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
|
||
`Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
|
||
- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
|
||
`STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
|
||
FP2=113), `ROCmI4`=118.
|
||
|
||
## 2. Comparison table
|
||
|
||
| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
|
||
|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
|
||
| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
|
||
| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
|
||
| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |
|
||
|
||
## 3. Code-difference narrative
|
||
|
||
**What each fork ADDED that mainline lacks:**
|
||
- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
|
||
format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
|
||
`*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
|
||
TurboQuant KV types, MTEA attention kernel. Mainline does not know these
|
||
tensor types. It is a *fork* that adds a new backend: custom tensor
|
||
registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
|
||
- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
|
||
CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
|
||
(`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
|
||
DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
|
||
`--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
|
||
`--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
|
||
(`-rtr`). **Additive** — GGUF remains compatible.
|
||
|
||
**What mainline has that the forks lag on:**
|
||
- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
|
||
architectures arrive late (PRs move fast, but the core is ~1.5y behind).
|
||
- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
|
||
fork line, CUDA ecosystem (the fleet's 3090) is unreachable.
|
||
|
||
**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
|
||
K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
|
||
CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
|
||
even where GPU would be faster → lower prompt-processing speed. For a MoE that
|
||
keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
|
||
MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
|
||
|
||
## 4. Server-specific benefits
|
||
|
||
### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
|
||
- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
|
||
MoE**, which is ik_llama.cpp's core strength:
|
||
- `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
|
||
RAM→VRAM traffic dramatically vs shipping all experts.
|
||
- `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
|
||
MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
|
||
- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
|
||
Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
|
||
K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
|
||
`ngram-mod` drafts (MoEs want long drafts).
|
||
- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
|
||
active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
|
||
the GGUF before finalizing the offload recipe — both point at expert-aware
|
||
offload being the right tool.
|
||
- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
|
||
- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.
|
||
|
||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
|
||
- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
|
||
ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
|
||
production `build-promptforge` `llama-server` already runs there.
|
||
- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
|
||
validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
|
||
draft-head speedup. Higher-risk change; only with a small soak-test first.
|
||
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
|
||
IU4, TurboQuant, and MTEA.
|
||
|
||
## 5. Recommendations for our fleet
|
||
|
||
- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
|
||
gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
|
||
2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
|
||
- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
|
||
`offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
|
||
much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
|
||
+ `ngram-mod`. Neither box needs ROCmFP4.
|
||
- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
|
||
change (FlashAttention + KV-quant are coupled).
|
||
|
||
## See Also
|
||
- [[Llama Cpp Runtimes]] — positioning + quantization overview
|
||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
|
||
- [[Quantization by Hardware Architecture]] — quant vs hardware
|
||
|
||
## Citations
|
||
- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
|
||
tools/quantize/README.md · docs/speculative.md
|
||
- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
|
||
- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, docs/parameters.md, docs/build.md) · DeepWiki · discussions#562
|
||
- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md
|