Frank
219a097b39
Every payload swap a live endpoint has ever run is now one row in that endpoint's Eras table, with its rollback path in the same row. No per-attempt page, no per-swap ledger, nowhere else. - Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork on the 3090). Each carries Current shape, Eras, Laws, wire contract, files. - Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras), the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing. - Rewired: every wikilink and markdown-path reference to a folded page points at the endpoint page instead; the three Research see-also lists and the ubatch sweep's runbook link included. No anchored links existed in the set. - Regenerated both index pages as pure locators -- one row, one clause, no numbers -- so the docs_completeness locator law holds for all 153 pages. tests/docs_completeness.bats green: index links every page, no unresolved links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored, and git history holds every byte).
142 lines
8.0 KiB
Markdown
142 lines
8.0 KiB
Markdown
---
|
||
type: research
|
||
title: Llama.cpp Runtimes
|
||
description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||
tags:
|
||
- llama.cpp
|
||
- quantization
|
||
- ROCmFPX
|
||
- ik_llama
|
||
- inference
|
||
- amd
|
||
- nvidia
|
||
timestamp: 2026-09-01
|
||
---
|
||
|
||
# Llama.cpp Runtimes
|
||
|
||
Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
|
||
the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
|
||
**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
|
||
(ikawrakow). The goal is to know, for each server, which quant formats are on
|
||
the table and which runtime best fits the hardware.
|
||
|
||
Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
|
||
positioning + quantization summary.
|
||
|
||
## Which runtime is which
|
||
|
||
| Runtime | Repo | What it is | Primary backends | For our boxes |
|
||
|---------|------|-----------|------------------|---------------|
|
||
| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
|
||
| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
|
||
| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |
|
||
|
||
Key lineage notes:
|
||
- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
|
||
commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
|
||
commit for reproducibility.
|
||
- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
|
||
three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
|
||
It is **AMD-only** — CUDA is compiled out.
|
||
- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
|
||
a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
|
||
appeared here first, later partially merged upstream). The feature surface is
|
||
current; the architecture baseline is ~1.5 years behind mainline.
|
||
|
||
## Quantization summary
|
||
|
||
### Mainline (what the fleet already ships)
|
||
- **Base:** FP32 / FP16 / BF16.
|
||
- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
|
||
`Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
|
||
`Q6_K` is "almost lossless."
|
||
- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
|
||
slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
|
||
family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
|
||
- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
|
||
- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
|
||
requires FlashAttention (`-fa on`).
|
||
- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).
|
||
|
||
### ROCmFPX (Dankclaw's AMD weight formats)
|
||
- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
|
||
**ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
|
||
(**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
|
||
`100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
|
||
Ollama, and LM Studio **cannot load them**.
|
||
- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
|
||
bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
|
||
units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
|
||
/ `*_STRIX_MTP` draft-head presets.
|
||
- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
|
||
(f16 K) on the production model.
|
||
- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
|
||
lossy activation quant, exact int8 MMQ.
|
||
|
||
### ik_llama.cpp (SOTA quants + CPU/hybrid)
|
||
- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
|
||
`Q4_K_M` at the same size. Improved **K-quants** and the signature
|
||
**row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
|
||
matmul paths that vanilla llama.cpp leaves cold).
|
||
- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
|
||
**Bitnet**.
|
||
- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
|
||
row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
|
||
tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
|
||
MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.
|
||
|
||
## Fit for our servers
|
||
|
||
### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
|
||
Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
|
||
GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
|
||
- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
|
||
shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
|
||
`--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
|
||
want long drafts).
|
||
- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
|
||
**offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
|
||
sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
|
||
fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
|
||
with the `-rtr` K-quant trap.
|
||
- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
|
||
The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.
|
||
|
||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
|
||
Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
|
||
- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
|
||
262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
|
||
- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
|
||
the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
|
||
- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
|
||
IU4, TurboQuant, MTEA, and agent presets.
|
||
|
||
## When to reach for which
|
||
- **Stay mainline** if you want one portable upstream for both boxes, the widest
|
||
GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
|
||
if you don't want the ik_llama.cpp build surface.
|
||
- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
|
||
justify an extra build (watch `-rtr` + K-quants).
|
||
- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
|
||
- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
|
||
port ROCmFP4 to the 3090 via Vulkan as a primary path.
|
||
|
||
## See Also
|
||
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
|
||
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
|
||
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
|
||
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
|
||
- [[SGLang]] — the structured-generation serving engine (the other lane)
|
||
|
||
## Citations
|
||
- Mainline — https://github.com/ggml-org/llama.cpp
|
||
- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
|
||
rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
|
||
https://github.com/charlie12345/rocmfp4
|
||
- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
|
||
https://ikawrakow-ik_llama-cpp.mintlify.app/
|
||
- ROCmFPX quant/enum/serve detail —
|
||
https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
|
||
- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/
|