Files

T

Frank 219a097b39 docs(vault): one page per endpoint, forever -- 45 era pages fold into three

Every payload swap a live endpoint has ever run is now one row in that
endpoint's Eras table, with its rollback path in the same row. No per-attempt
page, no per-swap ledger, nowhere else.

- Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md
  (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork
  on the 3090). Each carries Current shape, Eras, Laws, wire contract, files.
- Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the
  pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the
  benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras),
  the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus
  frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing.
- Rewired: every wikilink and markdown-path reference to a folded page points
  at the endpoint page instead; the three Research see-also lists and the
  ubatch sweep's runbook link included. No anchored links existed in the set.
- Regenerated both index pages as pure locators -- one row, one clause, no
  numbers -- so the docs_completeness locator law holds for all 153 pages.

tests/docs_completeness.bats green: index links every page, no unresolved
links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored,
and git history holds every byte).

2026-10-19 00:49:03 -07:00

8.0 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

Llama.cpp Runtimes

Comparison of three llama.cpp-derived inference runtimes used (or usable) on the two homelab boxes: mainline ggml-org/llama.cpp, charlie12345/ROCmFPX (what Dankclaw runs), and ik_llama.cpp (ikawrakow). The goal is to know, for each server, which quant formats are on the table and which runtime best fits the hardware.

Full code-level quantification is in Llama Cpp Code Diff. This page is the positioning + quantization summary.

Which runtime is which

Runtime	Repo	What it is	Primary backends	For our boxes
Mainline	`ggml-org/llama.cpp`	Canonical, general-purpose, one codebase for every backend	CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO	Baseline both boxes; the portable default
charlie12345/ROCmFPX	`charlie12345/ROCmFPX` (+ reference fork `rocmfp4-llama`, clean pkg `rocmfp4`)	Experimental AMD-focused GGUF weight quant family layered on llama.cpp	HIP/ROCm + Vulkan (CUDA disabled), CPU reference	Dankclaw (native Strix Halo)
ik_llama.cpp	`ikawrakow/ik_llama.cpp`	Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance	CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak)	Dankserver candidate (3090 + MoE)

Key lineage notes:

Mainline moved from ggerganov/llama.cpp to the ggml-org org. Rolling commit hashes (the fleet pins b10121) plus newer vX.Y.Z tags. Pin a commit for reproducibility.
ROCmFPX is an experimental feature family on the canonical main branch; three GitHub surfaces exist (reference fork, clean patch, pre-patched tree). It is AMD-only — CUDA is compiled out.
ik_llama.cpp's core is an August-2024 llama.cpp snapshot maintained with a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all appeared here first, later partially merged upstream). The feature surface is current; the architecture baseline is ~1.5 years behind mainline.

Quantization summary

Mainline (what the fleet already ships)

Base: FP32 / FP16 / BF16.
K-quants (modern default, two-level block + double-quant): Q2_K … Q6_K, with _S/_M/_L precision mixes. Q4_K_M is the popular default; Q6_K is "almost lossless."
i-quants (importance-matrix, non-linear, best quality/size at low bits but slower decode): IQ4_NL, IQ4_XS, IQ3_*, IQ2_*. Row-interleaved 3-bit family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
Legacy: Q*_0/Q*_1 (fast, simple), ternary TQ1_0.
KV cache quant: f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1; quantizing KV requires FlashAttention (-fa on).
Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, ngram-mod).

ROCmFPX (Dankclaw's AMD weight formats)

Integer GGUF weight formats only this fork reads: ROCmFP2/3/4/6/8 and ROCmI4 (W4A4 IU4 WMMA). These are custom ggml tensor types (ggml_type 100–108) with matching GGUF ftype (ROCmFP4 family 100–106; ROCmFPX family 110–113; ROCmI4 = 118). Stock llama.cpp, Ollama, and LM Studio cannot load them.
Flagship ROCmFP4 (block_rocmfp4): 32 weights/block, two E4M3 scale bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10 units. Variants: STRIX / STRIX_LEAN, FAST, COHERENT, plus *_AGENT / *_STRIX_MTP draft-head presets.
TurboQuant KV: Turbo3_0 (3.5 bpw) / Turbo4_0 (4.5 bpw) + TurboQuant V (f16 K) on the production model.
ROCmI4 (W4A4): the gfx1151 native IU4 4-bit WMMA path (-DGGML_HIP_ROCMI4_W4A4=ON); lossy activation quant, exact int8 MMQ.

ik_llama.cpp (SOTA quants + CPU/hybrid)

IQ* non-linear family; IQ4_NL gives lower perplexity than mainline Q4_K_M at the same size. Improved K-quants and the signature row-interleaved IQK / R4 / R8 packing (activates AVX-512/VNNI/NEON matmul paths that vanilla llama.cpp leaves cold).
Trellis quants (IQ*_KT), MXFP4, Q8_KV fast KV type, first-class Bitnet.
⚠️ K-quant / -rtr trap (critical for the 3090): K-quants have no CUDA row-interleaved path; -rtr (run-time repack) forces CPU-resident K-quant tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and let --fit/-ot place experts.

Fit for our servers

Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA

Serves Ornith-1.5-35B-A3B CRACK, a 35B MoE (~36 experts; the CRACK GGUF README lists 256/8-active — verify n_expert from the GGUF).

Mainline is the natural baseline: Q4_K_M/Q5_K_M, FlashAttention to shrink KV, GGML_CUDA_FORCE_MMQ=1 (default on RTX 30xx, lower VRAM), --cpu-moe to spill experts to the 62 GB RAM, and ngram-mod drafts (MoEs want long drafts).
ik_llama.cpp is the more compelling upgrade here: fused MoE (-fmoe), offload-only-active-experts (-ooae) which — with only ~8 active experts — sharply cuts RAM→VRAM traffic, --fit, smart-expert-reduction (-ser), GDN fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care with the -rtr K-quant trap.
ROCmFPX does not help a CUDA box (ROCm-only formats, CUDA disabled). The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4

Serves kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (the production build).

ROCmFPX is the native fit: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV, 262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs Q4_K_M).
ik_llama.cpp — ROCm is its weakest backend (gfx1151 not validated) and the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
Mainline ROCm build works for correctness (standard quants) but loses FPX, IU4, TurboQuant, MTEA, and agent presets.

When to reach for which

Stay mainline if you want one portable upstream for both boxes, the widest GGUF ecosystem, and standard FlashAttention/MTP. Best fit for Dankserver if you don't want the ik_llama.cpp build surface.
Use ik_llama.cpp for Dankserver when the MoE offload and fused-MoE wins justify an extra build (watch -rtr + K-quants).
Use ROCmFPX for Dankclaw — it is purpose-built for gfx1151.
ROCmFPX is for the AMD fleet; CUDA/FastMTP for the NVIDIA box. Do not port ROCmFP4 to the 3090 via Vulkan as a primary path.

Citations

Mainline — https://github.com/ggml-org/llama.cpp
charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX · rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 — https://github.com/charlie12345/rocmfp4
ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs · https://ikawrakow-ik_llama-cpp.mintlify.app/
ROCmFPX quant/enum/serve detail — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/

8.0 KiB Raw Permalink Blame History Unescape Escape