Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

7.9 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Llama.cpp Runtimes

Comparison of three llama.cpp-derived inference runtimes used (or usable) on the two homelab boxes: mainline ggml-org/llama.cpp, charlie12345/ROCmFPX (what Dankclaw runs), and ik_llama.cpp (ikawrakow). The goal is to know, for each server, which quant formats are on the table and which runtime best fits the hardware.

Full code-level quantification is in Llama Cpp Code Diff. This page is the positioning + quantization summary.

Which runtime is which

Runtime	Repo	What it is	Primary backends	For our boxes
Mainline	`ggml-org/llama.cpp`	Canonical, general-purpose, one codebase for every backend	CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO	Baseline both boxes; the portable default
charlie12345/ROCmFPX	`charlie12345/ROCmFPX` (+ reference fork `rocmfp4-llama`, clean pkg `rocmfp4`)	Experimental AMD-focused GGUF weight quant family layered on llama.cpp	HIP/ROCm + Vulkan (CUDA disabled), CPU reference	Dankclaw (native Strix Halo)
ik_llama.cpp	`ikawrakow/ik_llama.cpp`	Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance	CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak)	Dankserver candidate (3090 + MoE)

Key lineage notes:

Mainline moved from ggerganov/llama.cpp to the ggml-org org. Rolling commit hashes (the fleet pins b10121) plus newer vX.Y.Z tags. Pin a commit for reproducibility.
ROCmFPX is an experimental feature family on the canonical main branch; three GitHub surfaces exist (reference fork, clean patch, pre-patched tree). It is AMD-only — CUDA is compiled out.
ik_llama.cpp's core is an August-2024 llama.cpp snapshot maintained with a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all appeared here first, later partially merged upstream). The feature surface is current; the architecture baseline is ~1.5 years behind mainline.

Quantization summary

Mainline (what the fleet already ships)

Base: FP32 / FP16 / BF16.
K-quants (modern default, two-level block + double-quant): Q2_K … Q6_K, with _S/_M/_L precision mixes. Q4_K_M is the popular default; Q6_K is "almost lossless."
i-quants (importance-matrix, non-linear, best quality/size at low bits but slower decode): IQ4_NL, IQ4_XS, IQ3_*, IQ2_*. Row-interleaved 3-bit family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
Legacy: Q*_0/Q*_1 (fast, simple), ternary TQ1_0.
KV cache quant: f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1; quantizing KV requires FlashAttention (-fa on).
Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, ngram-mod).

ROCmFPX (Dankclaw's AMD weight formats)

Integer GGUF weight formats only this fork reads: ROCmFP2/3/4/6/8 and ROCmI4 (W4A4 IU4 WMMA). These are custom ggml tensor types (ggml_type 100–108) with matching GGUF ftype (ROCmFP4 family 100–106; ROCmFPX family 110–113; ROCmI4 = 118). Stock llama.cpp, Ollama, and LM Studio cannot load them.
Flagship ROCmFP4 (block_rocmfp4): 32 weights/block, two E4M3 scale bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10 units. Variants: STRIX / STRIX_LEAN, FAST, COHERENT, plus *_AGENT / *_STRIX_MTP draft-head presets.
TurboQuant KV: Turbo3_0 (3.5 bpw) / Turbo4_0 (4.5 bpw) + TurboQuant V (f16 K) on the production model.
ROCmI4 (W4A4): the gfx1151 native IU4 4-bit WMMA path (-DGGML_HIP_ROCMI4_W4A4=ON); lossy activation quant, exact int8 MMQ.

ik_llama.cpp (SOTA quants + CPU/hybrid)

IQ* non-linear family; IQ4_NL gives lower perplexity than mainline Q4_K_M at the same size. Improved K-quants and the signature row-interleaved IQK / R4 / R8 packing (activates AVX-512/VNNI/NEON matmul paths that vanilla llama.cpp leaves cold).
Trellis quants (IQ*_KT), MXFP4, Q8_KV fast KV type, first-class Bitnet.
⚠️ K-quant / -rtr trap (critical for the 3090): K-quants have no CUDA row-interleaved path; -rtr (run-time repack) forces CPU-resident K-quant tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/ MXFP4 variants, or skip -rtr and let --fit/-ot place experts.

Fit for our servers

Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA

Serves Ornith-1.5-35B-A3B CRACK, a 35B MoE (~36 experts; the CRACK GGUF README lists 256/8-active — verify n_expert from the GGUF).

Mainline is the natural baseline: Q4_K_M/Q5_K_M, FlashAttention to shrink KV, GGML_CUDA_FORCE_MMQ=1 (default on RTX 30xx, lower VRAM), --cpu-moe to spill experts to the 62 GB RAM, and ngram-mod drafts (MoEs want long drafts).
ik_llama.cpp is the more compelling upgrade here: fused MoE (-fmoe), offload-only-active-experts (-ooae) which — with only ~8 active experts — sharply cuts RAM→VRAM traffic, --fit, smart-expert-reduction (-ser), GDN fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care with the -rtr K-quant trap.
ROCmFPX does not help a CUDA box (ROCm-only formats, CUDA disabled). The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4

Serves kingjones Qwen3.8-27B ROCmFP4 STRIX MTP (the production build).

ROCmFPX is the native fit: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV, 262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs Q4_K_M).
ik_llama.cpp — ROCm is its weakest backend (gfx1151 not validated) and the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
Mainline ROCm build works for correctness (standard quants) but loses FPX, IU4, TurboQuant, MTEA, and agent presets.

When to reach for which

Stay mainline if you want one portable upstream for both boxes, the widest GGUF ecosystem, and standard FlashAttention/MTP. Best fit for Dankserver if you don't want the ik_llama.cpp build surface.
Use ik_llama.cpp for Dankserver when the MoE offload and fused-MoE wins justify an extra build (watch -rtr + K-quants).
Use ROCmFPX for Dankclaw — it is purpose-built for gfx1151.
ROCmFPX is for the AMD fleet; CUDA/FastMTP for the NVIDIA box. Do not port ROCmFP4 to the 3090 via Vulkan as a primary path.

Citations

Mainline — https://github.com/ggml-org/llama.cpp
charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX · rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 — https://github.com/charlie12345/rocmfp4
ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs · https://ikawrakow-ik_llama-cpp.mintlify.app/
ROCmFPX quant/enum/serve detail — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/

7.9 KiB Raw Blame History Unescape Escape