dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a5fe045b588b1240d3df4d20bce77f8491b19a03
serverAppConfigs/serverAppConfigs-docs/Research/Llama Cpp Runtimes.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

142 lines
7.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: reference
title: Llama.cpp Runtimes
description: "The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine."
tags:
- llama.cpp
- quantization
- ROCmFPX
- ik_llama
- inference
- amd
- nvidia
timestamp: 2026-09-01
---
# Llama.cpp Runtimes
Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
(ikawrakow). The goal is to know, for each server, which quant formats are on
the table and which runtime best fits the hardware.
Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
positioning + quantization summary.
## Which runtime is which
| Runtime | Repo | What it is | Primary backends | For our boxes |
|---------|------|-----------|------------------|---------------|
| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |
Key lineage notes:
- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
commit for reproducibility.
- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
It is **AMD-only** — CUDA is compiled out.
- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
appeared here first, later partially merged upstream). The feature surface is
current; the architecture baseline is ~1.5 years behind mainline.
## Quantization summary
### Mainline (what the fleet already ships)
- **Base:** FP32 / FP16 / BF16.
- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
`Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
`Q6_K` is "almost lossless."
- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
requires FlashAttention (`-fa on`).
- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).
### ROCmFPX (Dankclaw's AMD weight formats)
- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
**ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
(**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
`100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
Ollama, and LM Studio **cannot load them**.
- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
/ `*_STRIX_MTP` draft-head presets.
- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
(f16 K) on the production model.
- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
lossy activation quant, exact int8 MMQ.
### ik_llama.cpp (SOTA quants + CPU/hybrid)
- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
`Q4_K_M` at the same size. Improved **K-quants** and the signature
**row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
matmul paths that vanilla llama.cpp leaves cold).
- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
**Bitnet**.
- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.
## Fit for our servers
### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
`--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
want long drafts).
- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
**offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
with the `-rtr` K-quant trap.
- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
IU4, TurboQuant, MTEA, and agent presets.
## When to reach for which
- **Stay mainline** if you want one portable upstream for both boxes, the widest
GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
if you don't want the ik_llama.cpp build surface.
- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
justify an extra build (watch `-rtr` + K-quants).
- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
port ROCmFP4 to the 3090 via Vulkan as a primary path.
## See Also
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
- [[SGLang]] — the structured-generation serving engine (the other lane)
## Citations
- Mainline — https://github.com/ggml-org/llama.cpp
- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
https://github.com/charlie12345/rocmfp4
- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
https://ikawrakow-ik_llama-cpp.mintlify.app/
- ROCmFPX quant/enum/serve detail —
https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/
Reference in new issue View Git Blame Copy Permalink