dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): destroy the dead weight — 53 pages and the change diary

Browse Source 
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
This commit is contained in:
Frank
2026-10-19 10:56:37 -07:00
parent 8775e64ed0
commit 327e78b950
126 changed files with 443 additions and 14157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Research/AI Job Search and Resume Tools.md
+6 -6
View File
@@ -34,7 +34,7 @@ The space split into two distinct shapes in 2026:
your machine against a paid AI CLI (Claude Code, Codex, etc.).
**Strongest self-hostable picks for this fleet:** JobSync (MIT, Ollama-as-default
AI + MCP server), Resume Matcher (Apache-2.0, best dedicated tailoring tool),
AI + MCP server), Resume Matcher (Apache-2.0, best dedicated tailoring tool)
Reactive Resume (MIT, best pure resume builder, 43k stars). There is **no
dominant self-hosted app for pure job *discovery*** at scale — that niche is
JobNavigator (Docker, 126 stars) plus the CLI frameworks.
@@ -52,7 +52,7 @@ CLI you already pay for; data is local Markdown/YAML you own. Not a server:
`npx @santifer/career-ops init`.
### ai-job-search (MadsLorentzen) — https://github.com/MadsLorentzen/ai-job-search
MIT, ~43,600 stars. Claude Code framework: `/setup`, `/scrape`, `/rank`,
MIT, ~43,600 stars. Claude Code framework: `/setup`, `/scrape`, `/rank`
`/apply <url>` (fit evaluation + tailored LaTeX CV + cover letter + a reviewer
agent), `/interview`, plus an ATS parseability check on generated PDFs. Portal
skills ship for the Danish market but are swappable. Prereqs: Claude Code,
@@ -62,14 +62,14 @@ private repo with this repo as `upstream`.
### JobSync — https://github.com/Gsync/jobsync
MIT, ~1,300 stars. The most complete self-hosted tracker: application tracker +
monitoring dashboard, resume management (PDF/DOCX import with AI extraction),
monitoring dashboard, resume management (PDF/DOCX import with AI extraction)
contacts, company pages, interview question bank, tasks/time tracking, and
**automated job discovery** (scheduled automations track companies via job-board
APIs and AI-match each listing). In-app AI assistant (resume review, job-match
scoring, tailored cover letters) and a built-in **MCP server** (Claude Desktop
can add jobs from chat). **Ollama is the default provider — fully local, no API
key**; known-good `qwen3.5:9b`. Models must support tool calling + structured
output. `git clone` + `docker compose up` → :3737. Env: `AUTH_SECRET`,
output. `git clone` + `docker compose up` → :3737. Env: `AUTH_SECRET`
`ENCRYPTION_KEY` (encrypts stored API keys — set before adding keys, never
change after), `NEXTAUTH_URL`, `MCP_ENABLED`. Treats pasted postings as untrusted
text (prompt-injection defense).
@@ -115,7 +115,7 @@ highlighting** against the JD, PDF export (4 templates). **Ollama (local)** plus
OpenAI, Claude, Gemini, OpenRouter, DeepSeek (built on LiteLLM). Official
Docker images: `docker run -p 3000:3000 -v resume-data:/app/backend/data
ghcr.io/srbhr/resume-matcher:latest` (API at `/api`, health at `/api/v1/health`).
**Ollama-in-Docker gotcha (README):** use `http://host.docker.internal:11434`,
**Ollama-in-Docker gotcha (README):** use `http://host.docker.internal:11434`
not `localhost`.
### OpenResume — https://github.com/xitanggg/open-resume
@@ -133,7 +133,7 @@ DeepSeek, Groq) — no Ollama/local documented. Next.js 15 + Supabase + Redis.
### rendercv (non-AI) — https://github.com/rendercv/rendercv
MIT, ~17,600 stars. LaTeX-based YAML→PDF resume builder. No AI, but the
strongest "ATS-safe by construction" formatter (single column, standard fonts);
strongest "ATS-safe by construction" formatter (single column, standard fonts)
pairs well with any AI tailoring step.
## Best Practices for a Homelab AI Resume/Job Pipeline
serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md
+2 -2
View File
@@ -86,7 +86,7 @@ Three ways forward (way 1 applied 2026-08-19):
1. **Base a ComfyUI image on the AMD gfx1151 torch build.** Take `rocm/vllm:...gfx1151...pytorch_2.9.1...` (or AMD's PyTorch wheel index) and install ComfyUI on top. This is the direct path to GPU ComfyUI. **Applied:** the running `comfyui-rocm:gfx1151` image does exactly this (see `dankclaw/Containerfile.comfyui`).
2. **Build PyTorch for gfx1151** (`export PYTORCH_ROCM_ARCH=gfx1151` before build). Slow build, but a self-contained wheel afterwards.
3. **Use a ggml-family runtime instead** — stable-diffusion.cpp / Lemonade's `sd.cpp` backend ship their own gfx1151 HIP kernels and work today (see [[Lemonade and sd.cpp]]).
3. **Use a ggml-family runtime instead** — stable-diffusion.cpp / Lemonade's `sd.cpp` backend ship their own gfx1151 HIP kernels and work today (see ).
## Vulkan
@@ -111,6 +111,6 @@ podman rmi docker.io/rocm/vllm:rocm7.12.0_gfx1151_... #
## See Also
- [[ComfyUI]] — the service page (CPU config)
- [[Lemonade and sd.cpp]] — the ggml-family alternative with working gfx1151 kernels
- the ggml-family alternative with working gfx1151 kernels
- [[Observability]], [[Dankclaw Stack]]
- KB: [[ComfyUI]] (tools) (the "ROCm unavailable" claims here are now qualified)
serverAppConfigs-docs/Research/Lemonade and sd.cpp.md
-57
View File
@@ -1,57 +0,0 @@
---
type: reference
description: "Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot."
tags:
- research
timestamp: 2026-08-28
---
# Lemonade and sd.cpp
Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. Research from the Lemonade source repo (`lemonade-sdk/lemonade`, cloned 2026-08-14) and live verification.
## The short answer
Lemonade does not run ComfyUI. Its image-generation backend is a wrapped **stable-diffusion.cpp** (`leejet/stable-diffusion.cpp`) subprocess. sd.cpp is a ggml-family native C++ runtime that ships its own HIP kernels — and those kernels **include gfx1151**. ComfyUI runs on PyTorch, whose standard wheels omit gfx1151. Same hardware, different runtime, different result.
## From the Lemonade source
`src/cpp/server/backends/sdcpp/sdcpp_server.cpp`:
- Image gen is `SDServer`, a `WrappedServer` around a prebuilt sd.cpp binary.
- Downloads `sd-<version>-bin-Linux-Ubuntu-24.04-x86_64-<backend>.zip` from `leejet/stable-diffusion.cpp` releases.
- Backends: `cpu, cuda, metal, rocm, vulkan`.
- ROCm asset is `...-rocm-<version>.zip`; Vulkan is `...-vulkan.zip`.
Backend support matrix (`serverAppConfigs-docs/dev/backends-reference.md`):
| Recipe | Backend | Device families |
|--------|---------|-----------------|
| `sd-cpp` | rocm | amd_gpu (gfx103X, gfx110X, **gfx1150, gfx1151, gfx1152**, gfx120X) |
| `sd-cpp` | vulkan | amd_gpu; cpu; nvidia_gpu |
| `thenoise` | rocm | amd_gpu (gfx1150, gfx1151, gfx1152) — experimental |
**gfx1151 is explicitly supported** by sd.cpp's ROCm build. There is also a dedicated experimental `thenoise` ROCm backend targeting exactly the Strix Halo archs.
## Engine comparison
| | ComfyUI | Lemonade (sd.cpp) |
|---|---|---|
| Runtime | PyTorch | native C++ (ggml family) |
| AMD path | needs PyTorch ROCm wheel with gfx1151 kernels — not in standard wheels | ships own HIP kernels, gfx1151 included |
| On Strix Halo | CPU only (today) | ROCm GPU, or Vulkan fallback |
| Interface | node graph web UI | server, OpenAI-style API, GUI/CLI |
This mirrors llama.cpp on the same box: ggml-family runtimes ship their own gfx1151 HIP kernels and run on ROCm; PyTorch does not.
## Relevance
- If GPU image generation on Dankclaw matters, **Lemonade (or bare sd.cpp) is the working path today**.
- ComfyUI's node graph is the cost of switching: Lemonade is not ComfyUI.
- See [[ComfyUI ROCm and Vulkan]] for the PyTorch-side analysis showing a gfx1151-enabled torch would also make ComfyUI work.
## See Also
- [[ComfyUI ROCm and Vulkan]] — the PyTorch side of this question
- [[ComfyUI]] — current CPU config
- [[Dankclaw Stack]]
serverAppConfigs-docs/Research/Llama Cpp Code Diff.md
-222
View File
@@ -1,222 +0,0 @@
---
type: reference
title: Llama.cpp Code Diff
description: "Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards."
tags:
- llama.cpp
- quantization
- ggml
- ROCmFPX
- ik_llama
- code-diff
timestamp: 2026-09-01
---
# Llama.cpp Code Diff
Code-level diff of the three runtimes, focused on **quantization code** and the
differences that affect our two servers. Positioning overview is in
[[Llama Cpp Runtimes]].
## 1. Quantization file map
Where quant lives in each runtime (paths relative to the repo root):
| File / path | Mainline | ROCmFPX | ik_llama.cpp |
|-------------|----------|---------|--------------|
| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |
Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
`c49ebdbd`):
- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
`Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
`Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
`STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
FP2=113), `ROCmI4`=118.
## 2. Comparison table
| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |
## 3. Code-difference narrative
**What each fork ADDED that mainline lacks:**
- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
`*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
TurboQuant KV types, MTEA attention kernel. Mainline does not know these
tensor types. It is a *fork* that adds a new backend: custom tensor
registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
(`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
`--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
`--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
(`-rtr`). **Additive** — GGUF remains compatible.
**What mainline has that the forks lag on:**
- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
architectures arrive late (PRs move fast, but the core is ~1.5y behind).
- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
fork line, CUDA ecosystem (the fleet's 3090) is unreachable.
**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
even where GPU would be faster → lower prompt-processing speed. For a MoE that
keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
## 4. Server-specific benefits
### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
MoE**, which is ik_llama.cpp's core strength:
- `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
RAM→VRAM traffic dramatically vs shipping all experts.
- `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
`ngram-mod` drafts (MoEs want long drafts).
- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
the GGUF before finalizing the offload recipe — both point at expert-aware
offload being the right tool.
- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
production `build-promptforge` `llama-server` already runs there.
- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
draft-head speedup. Higher-risk change; only with a small soak-test first.
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
IU4, TurboQuant, and MTEA.
## 6. KV cache quant table
Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
**KV-native** (designed for the cache) or **weight-family reused for KV** (a
model-weight quant the runtime also accepts in `-ctk`/`-ctv`).
**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
This is the true in-RAM cost per value, so it is what sets KV-cache size and
quantize/dequant throughput. A footnote marks the few types whose **nominal**
figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
number; the table uses the exact number throughout.
### 6a. All KV-usable types, by bit density (interleaved)
| bpw | Type | Origin | KV-native | Notes |
|-----|------|--------|-----------|-------|
| 32.0 | f32 | mainline * | yes | full precision |
| 16.0 | f16 | mainline * | yes | default |
| 16.0 | bf16 | mainline * | yes | |
| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
| 8.13 | **q8_0** | mainline * | yes | |
| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
| 6.63 | iq6_K | ik | weight→KV | IQK family |
| 6.56 | q6_K | mainline * | weight→KV | |
| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
| 5.50 | q5_K | mainline * | weight→KV | |
| 5.50 | iq5_K | ik | weight→KV | IQK family |
| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
| 4.50 | iq4_K | ik | weight→KV | IQK family |
| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
| 4.25 | mxfp4 | ik * | weight→KV | |
| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
| 3.44 | iq3_K | ik | weight→KV | IQK family |
| 3.44 | iq3_s | mainline * | weight→KV | |
| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
| 2.38 | iq2_K | ik | weight→KV | IQK family |
| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |
### 6b. Origin legend
- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
`kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
with `*` and `yes` in the KV-native column.
- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
(add a third `V` type for values only). Marked `*` here.
- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
`IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
`mxfp4`. ik's `-ctk` accepts a broader set than mainline's.
### 6c. Trellis, IQK, MXFP4 — where they sit
- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
quality per bit of the family; lowest-density entries in the whole table.
- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
`iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
`iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
(iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.
### 6d. Hadamard is a transform, not a row
`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
applied on top of a base KV quant**, not its own bit-density row. It spreads
quantization error, improving accuracy for the underlying type; ik's guidance is
**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
a packing transform on the K-family — and is likewise a modifier, not a quant.
### 6e. Server notes
- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
`turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
Strix is higher-risk (see §4).
- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
— they are coupled (a quant change that drops FA off the path changes KV math).
## 5. Recommendations for our fleet
- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
`offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
+ `ngram-mod`. Neither box needs ROCmFP4.
- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
change (FlashAttention + KV-quant are coupled).
## See Also
- [[Llama Cpp Runtimes]] — positioning + quantization overview
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — quant vs hardware
## Citations
- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
tools/quantize/README.md · serverAppConfigs-docs/speculative.md
- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, serverAppConfigs-docs/parameters.md, serverAppConfigs-docs/build.md) · DeepWiki · discussions#562
- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md
serverAppConfigs-docs/Research/Llama Cpp Runtimes.md
-141
View File
@@ -1,141 +0,0 @@
---
type: reference
title: Llama.cpp Runtimes
description: "The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine."
tags:
- llama.cpp
- quantization
- ROCmFPX
- ik_llama
- inference
- amd
- nvidia
timestamp: 2026-09-01
---
# Llama.cpp Runtimes
Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
(ikawrakow). The goal is to know, for each server, which quant formats are on
the table and which runtime best fits the hardware.
Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
positioning + quantization summary.
## Which runtime is which
| Runtime | Repo | What it is | Primary backends | For our boxes |
|---------|------|-----------|------------------|---------------|
| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |
Key lineage notes:
- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
commit for reproducibility.
- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
It is **AMD-only** — CUDA is compiled out.
- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
appeared here first, later partially merged upstream). The feature surface is
current; the architecture baseline is ~1.5 years behind mainline.
## Quantization summary
### Mainline (what the fleet already ships)
- **Base:** FP32 / FP16 / BF16.
- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
`Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
`Q6_K` is "almost lossless."
- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
requires FlashAttention (`-fa on`).
- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).
### ROCmFPX (Dankclaw's AMD weight formats)
- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
**ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
(**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
`100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
Ollama, and LM Studio **cannot load them**.
- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
/ `*_STRIX_MTP` draft-head presets.
- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
(f16 K) on the production model.
- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
lossy activation quant, exact int8 MMQ.
### ik_llama.cpp (SOTA quants + CPU/hybrid)
- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
`Q4_K_M` at the same size. Improved **K-quants** and the signature
**row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
matmul paths that vanilla llama.cpp leaves cold).
- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
**Bitnet**.
- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.
## Fit for our servers
### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
`--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
want long drafts).
- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
**offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
with the `-rtr` K-quant trap.
- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
IU4, TurboQuant, MTEA, and agent presets.
## When to reach for which
- **Stay mainline** if you want one portable upstream for both boxes, the widest
GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
if you don't want the ik_llama.cpp build surface.
- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
justify an extra build (watch `-rtr` + K-quants).
- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
port ROCmFP4 to the 3090 via Vulkan as a primary path.
## See Also
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
- [[SGLang]] — the structured-generation serving engine (the other lane)
## Citations
- Mainline — https://github.com/ggml-org/llama.cpp
- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
https://github.com/charlie12345/rocmfp4
- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
https://ikawrakow-ik_llama-cpp.mintlify.app/
- ROCmFPX quant/enum/serve detail —
https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/
serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
-467
View File
@@ -1,467 +0,0 @@
---
type: reference
title: Local TTS and Voice Cloning
description: "Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack."
tags:
- research
- tts
- voice-cloning
- audio
- stt
- vllm-omni
- kokoro
timestamp: 2026-10-18T16:05:00Z
---
# Local TTS and Voice Cloning
Research brief (2026-09-22) on creating a new local TTS voice from sample
files, the open-weight voice-cloning landscape, the full local AI audio
stack, and a recommended deployment for the fleet. Every claim is cited in
[Citations](#citations). The fleet has no audio input today (that capability
left with the retired NPU endpoint, returned with the Gemma4 audio projector on
2026-09-17, and left again with that retirement on 2026-10-01 — audio is still
greenfield: every fleet endpoint declares `input: [text, image]`
(`~/.omp/agent/models.yml:63,149,214`).
## Summary
- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
Voices are a fixed pack. A community fine-tune recipe exists
(`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
the output is Kokoro-styled, and OMP's local backend only accepts its
built-in voice list (a custom pack needs a patch).
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
(Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT weights)**,
**VoxCPM2 (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration (re-verified 2026-10-18):** the speech tool is gated by
`speechgen.enabled` and resolves models through the **`speech` role and its
fallback chain** (`modelRoles.speech`; this workstation runs
`local/kokoro`). Three APIs: local Kokoro ONNX (always WAV/PCM16), `xai-tts`,
and **`openai-speech` — a generic OpenAI-compatible `/v1/audio/speech`
client** (MP3 or WAV, `voice_id` forwarded only when explicitly set, 15 000
characters max; DeepInfra is now just a model on that API). The 2026-09-22
finding that OMP has *no* configurable OpenAI TTS endpoint is **retired**: a
local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
#2 rules out every GPU option (the 3090 has carried frank-zealot since
2026-10-17; both R9700s are frank-archon's). The shape is a TTS engine on
**8880** behind **frank-tts**, the fleet's speech endpoint on **7772**
(`scripts/frank-tts.env`, `dankclaw/voiceservice/`): one OpenAI
`/v1/audio/speech` façade that resolves a voice *by name* and never shows a
path. Nothing is deployed — no `frank-tts` unit and no container exist on the
host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet
TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine is chosen: arm A**
`qwentts.cpp` **resident `tts-server`, 1.7 B talker** — measured in
*Measured CPU engine A/B* below and picked on the live-output criterion in
[Decision: arm A for live output](#decision-arm-a-for-live-output-2026-10-18).
## Can Kokoro learn a new voice?
Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
params, non-autoregressive) ships **fixed voice packs** — there is no
reference-audio path, so zero-shot cloning is not an option. OMP's local
backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
from a built-in list (`tts.localVoice`).
The best-documented fine-tune path is
[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):
| Stage | Data | Purpose |
|---|---|---|
| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
| Extraction | — | voice-pack (.pt) extraction for inference |
**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
and patch OMP to load the custom pack. The effort/quality trade is worse than
running a modern zero-shot cloner on the same GPU.
## 2026 voice-cloning landscape (permissive-licensed, single-GPU)
Ranked (2026-09-22) for a 3090 homelab, English-primary, personal voice; the
2026-10-18 fleet decision below moved this work off the GPU entirely:
1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference
(the README's own example is `your_10s_ref_clip.wav`); 110 M (Nano) to 0.5 B
(Turbo/Multilingual); ~1–2 GB VRAM *(unverified 2026-10-18)*; Nano is
documented for CPU — "3x faster than realtime on 8 CPU cores". *Top pick for
a personal English voice.* No server in the README (library use only; wrap
it yourself); every generated file carries Resemble's PerTh watermark.
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
vLLM-Omni serving with a `POST /v1/audio/voices` upload *(unverified
2026-10-18 — the reachable docs do not list it)*, the closest thing to
"named voice" semantics, official fine-tuning code. *Top pick for a
OpenAI-compatible deployment.*
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
bi-streaming, FastAPI and gRPC servers.
4. **GLM-TTS (MIT model weights, Apache-2.0 code)** — CER 0.89 after
multi-reward GRPO (down from 1.03), RL emotion control, zh/en; the prompt
audio shipped in the repo is marked research-only.
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
Docker images.
Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
encoder not released); Voxtral TTS (no official cloning — codec encoder not
released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
(code withdrawn 2025-09).
## OMP integration (definitive)
The tool ships hidden and is enabled by `speechgen.enabled`
(`~/.omp/agent/config.yml:222-223`); the harness docs list it as "Generate
speech". Models come from the **`speech` role and its fallback chain**, not
from a hardcoded backend list (`modelRoles.speech`,
`~/.omp/agent/config.yml:16` = `local/kokoro` today):
| API | Endpoint | Notes (from the installed bundle, 2026-10-18) |
|---|---|---|
| local-inference | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` (Apache-2.0) | no network, **always WAV/PCM16**, voice from the built-in list (`tts.localVoice`) |
| `xai-tts` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + custom voice IDs; MP3 or WAV |
| `openai-speech` | `POST <provider baseUrl>/audio/speech` | the generic OpenAI-compatible client — MP3 or WAV, `voice_id` forwarded **only when explicitly set**, 15 000-character cap on `input`; DeepInfra is one model entry on it (`https://api.deepinfra.com/v1/openai`) |
The 2026-09-22 finding — "**no generic OpenAI-compatible TTS endpoint** …
`deepinfra` is a hardcoded URL, not a configurable base URL" — is **retired**:
`openai-speech` is that configurable client, selected by putting a model on the
`speech` role. Still true from the original audit: the local Kokoro backend
accepts only its built-in voice list, so a custom pack needs a patch.
Paths for a Frank-cloned voice (2026-10-18):
1. **Local server + speech role, no patch.** Register frank-tts (port 7772) as
a provider model on the `openai-speech` API and point `modelRoles.speech` at
it; the façade turns a voice *name* into that pack's reference clip and
forwards it as base64 `reference_audio` (`scripts/frank-tts.env`,
`TTS_BACKEND=omni-http`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
tool. Free (≤30 voices), cloud, US-only.
3. **No-patch workaround.** The `bash` tool can `curl` the local TTS server ad
hoc, and the harness `tts` device writes a speech file to a path you give it.
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.
## Full local AI audio stack
**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
60-min, 50+ langs).
**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
faster-whisper/whisperX).
**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
Apache-2.0) publishes the official images `vllm/vllm-omni` (CUDA) and
`vllm/vllm-omni-rocm` (ROCm), both tagged `v0.28.0` (Docker Hub tag lists,
2026-10-18), and serves Qwen3-TTS (`docs/models/supported_models.md`). It
serves the OpenAI speech path `POST /v1/audio/speech`; the
`POST /v1/audio/voices` upload route is *(unverified 2026-10-18)* — the docs
reachable today do not document it. Its gfx942/gfx950 validation note is a
footnote scoped to MiniMax H3 diffusion, **not** to TTS, and the fleet runs no
vLLM anywhere: the R9700s carry radiance as frank-archon
(`scripts/serve-frank-archon.sh:2`). It is also no longer "the one server" with
the OpenAI shape — llama.cpp's own `llama-tts-server` has it too (next
section). Others: Kokoro-FastAPI (Docker, :8880, Apache-2.0), CosyVoice
**FastAPI and gRPC** (`runtime/python/fastapi`, `runtime/python/grpc`; the
README documents no WebSocket server), F5-TTS Gradio/Triton, GPT-SoVITS
WebUI/api_v2, Fish S2 `tools/api_server.py` (own format, not OpenAI-shaped).
**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
cloning; the Perth watermark is built into Chatterbox.
## llama.cpp-native TTS serving (verified 2026-10-18)
The CPU shape does not need a Python server at all:
- **`llama-tts-server`** ([`tc-mb/llama.cpp-omni`](https://github.com/tc-mb/llama.cpp-omni),
MIT, "Omni inference in C/C++") is a standalone OpenAI-compatible TTS server
that runs without an LLM model (build target `llama-tts-server`).
`POST /v1/audio/speech` takes `model` (`"voxcpm"`, also accepts `"voxcpm2"`),
`input`, `voice` — documented **"reserved"**, so do not build named-voice
semantics on it — `response_format` (`wav`/`pcm`), `reference_audio`
(base64 WAV for cloning), `seed` 42, `cfg_value` 2.0, `inference_timesteps`
10, `max_steps` 200, `temperature` 1.0. `POST /v1/audio/speech/stream`
returns chunked WAV; `POST /v1/voxcpm2/init` hot-loads a runtime from
`base_lm` + `acoustic` GGUFs with `n_gpu_layers` (`-1` = all);
`GET /v1/audio/speech/models` lists what is loaded
(`tools/omni/voxcpm2/README.md`:232, 321-354).
- **Mainline `llama-tts`** (ggml-org/llama.cpp) has `--tts-speaker-file` for a
speaker reference with **Qwen3-TTS** and with **Kyutai Pocket-TTS** — the
latter requires it ("the model produces almost no audio without it"), and
Pocket-TTS converts one `languages/<name>` directory, never the repo root
(`tools/tts/README.md`:20-58; `kyutai/pocket-tts` is CC-BY-4.0).
- **VoxCPM2 GGUF provenance:** `openbmb/VoxCPM2` publishes **PyTorch weights
only** (`model.safetensors`, no `.gguf` sibling — HF API 2026-10-18). The
GGUFs the fork loads come from
[`DennisHuang648/VoxCPM2-GGUF`](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF)
(`VoxCPM2-BaseLM-F16.gguf`, `VoxCPM2-BaseLM-Q8_0.gguf`,
`VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
`tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.
Arm B (`llama-tts`) and arm A (`qwentts.cpp`) have both been measured on this
box — RTF, first-audio latency and RSS are in the next section. The other
servers in the table above (vLLM-Omni, Kokoro-FastAPI, CosyVoice, F5-TTS,
GPT-SoVITS) remain unmeasured here; that absence is still the finding.
## Measured CPU engine A/B (2026-10-18)
Both candidate upstreams for frank-tts were measured on Dankclaw's CPU — 8
threads, no GPU touched, three runs per arm, seed 42, identical text, against
the fleet's own audition voice packs.
- **Arm A — [`ServeurpersoCom/qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp)**
at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (2026-10-16), weights
from [`Serveurperso/Qwen3-TTS-GGUF`](https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF):
`qwen-talker-1.7b-base-Q4_K_M.gguf` (1,219,245,248 B),
`qwen-talker-0.6b-base-Q4_K_M.gguf` (628,905,056 B),
`qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B) — byte counts verified
against the Hub API. Builds with zero compiler warnings; `GGML_BLAS=OFF` is
forced on this host (only `libopenblas.so.0` exists, no dev headers, so the
vendor `buildcpu.sh` cannot configure).
- **Arm B — mainline `llama-tts`** with `--tts-speaker-file`, i.e. the
`llama-tts-cli` backend `scripts/frank-tts.env` already implements.
RTF is **seconds of compute per second of audio** (lower is better; below 1 is
faster than real time).
| Arm | RTF | Run spread | Peak RSS |
|---|---|---|---|
| A · 0.6 B talker, CLI | 0.44 | ±0.002 | 3.4 GiB |
| A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB |
| A · 1.7 B, CLI, embedding-only clone | 0.63 | ±0.02 | n/m |
| A · 1.7 B, CLI, ICL clone | 0.92 | ±0.01 | n/m |
| A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB |
| A · same, ICL voice warm / **cold (first request)** | 0.67 / **1.65** | one-shot | 5.7 GiB |
| B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m |
**Read RTF only within an arm.** Conditioning changes clip length — the same
seed yields 18.00 s of audio embedding-only and 15.12 s under ICL — so the
denominator is the arm's own audio duration.
Arm A's resident server is ~1.7× faster than arm B and about 40× tighter in
run-to-run spread, and it already serves the HTTP shape frank-tts's façade
speaks: `/health`, `/v1/models`, `/v1/audio/voices` (GET/POST/DELETE),
`/v1/audio/speech` (`wav`/`pcm`), chunked streaming with the first audio byte
at 86–451 ms and a server-side time-to-first-audio of 94–112 ms. Arm B has no
HTTP path at all — that is what the façade's CLI backend exists for.
Four laws came out of the trial, each worth a line of configuration:
- **Boot-warm every voice.** The trial measured the first request for a
registered voice at 6.2–8.0 s (first audio byte 5,530 ms) and later requests
restoring the snapshot in 0.5 ms. **That ladder did not survive deployment:**
measured 2026-10-18 on the built container against an empty registry — register,
then time — the first synthesis of a freshly registered voice cost **1,883 ms**
against **1,226–1,283 ms** for the requests after it, and a ~110-word request
cost **9.4 s** then **9.8 s**, so request time tracks the audio generated rather
than snapshot state. What still holds is the reason to warm at all: the registry
is memory-only and re-registering does not warm, so an engine unit must run one
throwaway synthesis per voice at start. It buys registration and about 0.6 s,
not five seconds.
- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference
WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T,
11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a
**byte-identical** WAV and saves 1.4 s per request. That pair is the
engine-artifact term the voice-pack law in `scripts/frank-tts.env` reserves a
slot for — arm A pins it as `.spk` + `.rvq` beside `reference.wav`.
- **Streaming is not bit-identical to buffered**, whatever the upstream README
claims: same text, seed and voice, 92.2 % of samples differ (peak delta
1139/32768, −41.6 dB RMS — inaudible). A fixed seed does not pin output
across `response_format`, so never mix the two paths in a cache or a golden
fixture.
- **A voice cannot be registered from an embedding alone**: `POST
/v1/audio/voices` accepts `wav_b64` or **both** `spk_b64` and `rvq_b64`
(`src/tts-server.h:480`), so an embedding-only voice has to be sent as the
WAV, or the lone-`spk` path has to be upstreamed.
**Likeness is not measurable at this resolution.** x-vector cosine against each
speaker's own reference: ICL 0.9886 / 0.9901, embedding-only 0.9844 / 0.9812,
uncloned 0.9228 — but **two different speakers score 0.9497** on the same
metric. The whole clone effect lives in a 0.04-wide band and the
ICL-over-embedding gap is inside noise, so the four audition clips
(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3`
mainline partner, guide in `LISTEN-engine.md`) are the instrument. **The
operator heard them on 2026-10-18 and passed every clone**, which removed the
last axis measurement could not reach; the engine was then picked on
live-output grounds ([Decision](#decision-arm-a-for-live-output-2026-10-18)).
The cosine
extractor is still worth keeping: it is engine-neutral (it scores mainline
output through the same encoder) and it separates an uncloned voice cleanly.
Two operational details. The engine takes 8 threads and exposes **no `-t`
flag** — `src/backend.h` uses `hardware_concurrency()/2` and prints the choice
at load, so pinning a cpuset on the quadlet changes that number implicitly. And
ICL needs a reference transcript that did not exist: the fleet has no ASR
endpoint and no audio input, so the crops were transcribed with
`vosk-small-en-us` inside the work dir, and two of three transcripts came back
suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into
the pack manifest instead of transcribing on the host.
## Decision: arm A for live output (2026-10-18)
Every audition pair passed, so likeness stopped being the tiebreak. The
question became **which engine is better for live output** — the harness
speaking a reply, an alert read aloud, a click-to-speak — and on that criterion
RTF only has to stay under 1.0 with margin: what separates the arms is **time to
first audio** and the **spread**, because an arm whose RTF can cross 1.0 stalls
a live stream mid-sentence.
| | arm A · `qwentts.cpp` `tts-server` | arm B · mainline `llama-tts` |
|---|---|---|
| first audio byte | 86–451 ms | model load, per request |
| warm RTF | 0.55–0.67 | 0.93–0.99 |
| run spread | ±0.01 | ±0.25 |
| HTTP | serves `/v1/audio/speech` | none — façade CLI only |
| RAM | 5.7 GiB resident | per-request, freed after |
**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine`
container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.**
Three consequences are configuration, not opinion: the unit **boot-warms every
voice** — the registry is memory-only, so an unwarm'd name is a 400 and the first
synthesis otherwise costs about 0.6 s more than the ones after it, as corrected
above —, it **does not pin a cpuset** casually (threads are
`hardware_concurrency()/2`, no flag), and every pack registers with the **WAV**
because the engine rejects a lone embedding.
Two follow-ups stay open on purpose, and neither blocks the deploy:
- **Talker size.** The 0.6 B arm measured 0.44 RTF at 3.4 GiB and has never
been auditioned — every clip above is 1.7 B. It is the lever for a CPU
co-tenancy problem, not a preemptive saving.
- **ICL vs embedding-only.** Both passed the ear test. Embedding-only is the
shipped default: 0.57–0.63 vs 0.67 warm, and it needs no `ref_text`, which
this fleet cannot produce honestly (the ASR crops came back suspect). Packs
still carry `.spk` + `.rvq`, so ICL remains a per-voice opt-in for batch work
like the podcast render, where total time matters more than first byte.
## Recommended deployment (Dankclaw)
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus one
façade, and **nothing here is deployed as of 2026-10-18**: `frank-tts` has
neither a unit nor a container on the host, and nothing listens on 7772 or
8880.
| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS engine | **arm A `qwentts.cpp` `tts-server`** on the Qwen3-TTS GGUFs (decided 2026-10-18, above); arm B mainline `llama-tts` stays in the repo as the `llama-tts-cli` rollback; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 (container-internal) | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself and boot-warms each voice; arm B has no HTTP path and runs through the façade's CLI backend |
| frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices/<name>/reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
7772 is frank-tts's host port — the next slot in the fleet's model-endpoint
family (7770 frank-dragoon, 7771 frank-zealot). The engine behind it keeps 8880,
but **not as a host-loopback publish**: netavark DNATs published ports only, and
a bridge container cannot reach a `127.0.0.1` bind on the host — that is the
lesson that 502'd hermes (`AGENTS.md`). `host.containers.internal`
(169.254.1.2) is the gateway, so it reaches a host service only when that
service binds a routable address; `scripts/frank-tts.env` names it as the
engine's address, which holds for a `0.0.0.0` bind and not for a loopback one.
The deployed pair therefore shares one quadlet network and the façade addresses
the engine by container name.
## Voice-creation workflow (the "from sample files" answer)
1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
GLM-TTS clone from it per request (no training).
3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
(vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
backend — highest effort, Kokoro-styled output.
## Best practices
- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
fine-tune on quality and effort.
- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
clean voice with Demucs if the source is music/mixed.
- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
for many TTS models — it is the only server with the standard
`/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
- **Check the license before serving publicly:** several strong cloners are
non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
- **Keep TTS off the GPUs.** The 2026-09-22 advice to "co-locate on the 3090 at
`--gpu-memory-utilization ~0.35`" is retired: that card belongs to
frank-zealot (2026-10-17) and the R9700s to frank-archon, and Gitea issue #2
measures both as full. Run TTS on CPU; revisit only if the operator frees a
card on purpose.
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
internal address, never the public hostname.
## Citations
- OMP speech tool: https://omp.sh/docs/tools ("Generate speech", gated by
`speechgen.enabled`; the old `https://omp.sh/docs/tools/tts` path returns 404
as of 2026-10-18) · installed binary
`@oh-my-pi/pi-coding-agent/dist/cli.js` (the `openai-speech` API, the
15 000-character cap, the `voice_id` forwarding rule, local Kokoro WAV-only)
- Kokoro: https://github.com/hexgrad/Kokoro ·
https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
- Chatterbox: https://github.com/resemble-ai/chatterbox
- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
https://huggingface.co/zai-org/GLM-TTS
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- qwentts.cpp (arm A): https://github.com/ServeurpersoCom/qwentts.cpp at commit
`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` · weights
https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF (byte counts against the
Hub API, 2026-10-18)
- mainline `llama-tts`:
https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)
- VoxCPM2 GGUFs (not published by OpenBMB):
https://huggingface.co/DennisHuang648/VoxCPM2-GGUF
- GLM-TTS license split: model card `mit` (HF API 2026-10-18) vs code
`LICENSE` (Apache-2.0):
https://raw.githubusercontent.com/zai-org/GLM-TTS/main/LICENSE
- F5-TTS: https://github.com/SWivid/F5-TTS
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
- IndexTTS: https://github.com/index-tts/index-tts
- Fish Speech: https://github.com/fishaudio/fish-speech
- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
- XTTS: https://github.com/coqui-ai/TTS
- MegaTTS3: https://github.com/bytedance/MegaTTS3
- Higgs Audio: https://github.com/boson-ai/higgs-audio
- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
- Parler-TTS: https://github.com/huggingface/parler-tts
- VibeVoice: https://github.com/microsoft/VibeVoice
- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
https://docs.vllm.ai/projects/vllm-omni/en/latest/
- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
whisperX: https://github.com/m-bain/whisperX
- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
Canary: https://huggingface.co/nvidia/canary-1b-v2
- Moonshine: https://github.com/moonshine-ai/moonshine
- Silero VAD: https://github.com/snakers4/silero-vad
- xAI custom voices:
https://docs.x.ai/developers/model-capabilities/audio/custom-voices
serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md
-218
View File
@@ -1,218 +0,0 @@
---
type: reference
description: "The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes."
tags:
- research
- self-hosted
- ai-apps
- landscape
timestamp: 2026-09-22
---
# Most Popular Self-Hosted AI Apps (June–September 2026)
Research brief on the most popular and trending self-hostable (Docker/Podman/
Quadlet) open-source AI applications over the last 3 months. Star counts and
licenses pulled directly from the GitHub API and each repo's LICENSE file on
**2026-09-22** (exact values). "Trending" is evidenced by **release activity
dated June–September 2026** (GitHub releases API) plus press/community
coverage — GitHub does not expose star-growth rate, so cumulative stars are the
popularity proxy and release cadence is the trend signal. The full rendered
report lives at `popular-self-hosted-ai-apps.html` beside this page.
## Summary
The self-hosted AI stack in mid-2026 consolidated into a few very large, very
active projects, and the center of gravity shifted hard toward **agents** over
the last quarter:
1. **Agents are the category of 2026.** OpenClaw (390,235 stars, MIT) is the
single most-starred self-hosted AI project and the fastest-growing repo on
GitHub; it shipped a major **2.0 rewrite on 1 September 2026** (933
contributors, 16,000+ PRs). Nous Research's **Hermes Agent** (247,907 stars,
MIT) is a close second and releases weekly. Both are model-agnostic
orchestration layers that run on your own hardware and can point at local
llama.cpp/vLLM/Ollama endpoints.
2. **The inference + UI layer is mature and huge.** Ollama (181,444), llama.cpp
(129,149), and vLLM (92,405) are the runtimes; **Open WebUI** (152,784) is
the dominant chat UI; **ComfyUI** (134,407, GPL-3.0) is the dominant
image/video-gen GUI. All release frequently in the window.
3. **RAG/agent-platforms and the audio stack round out the field.** Dify
(156,835), RAGFlow (91,138), AnythingLLM (66,318) for RAG/workflows;
whisper.cpp (53,852) + faster-whisper (25,512) for STT; Chatterbox (26,515),
Piper (11,287), Kokoro-82M (8,933) for TTS; Unsloth (76,560) for
fine-tuning.
**Strongest fits for this fleet:** Open WebUI (chat UI over the existing
llama.cpp/vLLM endpoints), ComfyUI (image gen, already integrated with Open
WebUI), Ollama (OpenAI-compatible runtime), Dify and RAGFlow (RAG/agent
platforms), and whisper.cpp / Piper / Kokoro for a local voice stack. OpenClaw
and Hermes are the highest-buzz new arrivals, well-suited to a homelab server,
with the caveat that they are tool/shell-access agents and need the strongest
isolation.
## Ranked List (Top 15, all with June–Sept 2026 release activity)
| # | App | Category | Stars (2026-09-22) | License | Latest release in window |
|---|-----|----------|--------------------|---------|--------------------------|
| 1 | [OpenClaw](https://github.com/openclaw/openclaw) | Agents | 390,235 | MIT | v2026.9.5 (09-19); 2.0 (09-01) |
| 2 | [Hermes Agent](https://github.com/NousResearch/hermes-agent) | Agents | 247,907 | MIT | v2026.9.21 (09-21) |
| 3 | [Ollama](https://github.com/ollama/ollama) | LLM runtime | 181,444 | MIT | active (pushed 09-21) |
| 4 | [Dify](https://github.com/langgenius/dify) | Agent platform / RAG | 156,835 | Modified Apache-2.0 | 1.17.1 (09-10) |
| 5 | [Open WebUI](https://github.com/open-webui/open-webui) | Chat UI | 152,784 | Open WebUI License (BSD-3-based) | v0.11.4 (09-21) |
| 6 | [ComfyUI](https://github.com/Comfy-Org/ComfyUI) | Image / video gen | 134,407 | GPL-3.0 | v0.37.0 (09-21) |
| 7 | [llama.cpp](https://github.com/ggml-org/llama.cpp) | LLM runtime (inference) | 129,149 | MIT | active (pushed 09-22) |
| 8 | [vLLM](https://github.com/vllm-project/vllm) | LLM inference engine | 92,405 | Apache-2.0 | active (pushed 09-22) |
| 9 | [RAGFlow](https://github.com/infiniflow/ragflow) | RAG engine | 91,138 | Apache-2.0 | v0.27.2 (09-10) |
| 10 | [LobeChat](https://github.com/lobehub/lobehub) | Chat UI / agent ops | 82,748 | LobeHub Community License | active (pushed 09-22) |
| 11 | [Unsloth](https://github.com/unslothai/unsloth) | Fine-tuning | 76,560 | Apache-2.0 | active (pushed 09-22) |
| 12 | [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) | RAG / local agent | 66,318 | MIT | active (pushed 09-21) |
| 13 | [whisper.cpp](https://github.com/ggml-org/whisper.cpp) | Speech-to-text | 53,852 | MIT | active (pushed 09-22) |
| 14 | [LibreChat](https://github.com/danny-avila/LibreChat) | Chat UI | 44,608 | MIT | active (pushed 09-22) |
| 15 | [Tabby](https://github.com/TabbyML/tabby) | AI coding assistant | 33,889 | Apache-2.0 (ee/ separate) | last push 2026-06-30 |
**Audio honorable mentions:** [Chatterbox](https://github.com/resemble-ai/chatterbox)
(26,515, MIT, TTS), [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
(25,512, MIT, STT), [Piper TTS](https://github.com/rhasspy/piper) (11,287, MIT,
TTS), [Kokoro-82M](https://github.com/hexgrad/kokoro) (8,933, Apache-2.0, TTS),
[MisoTTS](https://github.com/MisoLabsAI/MisoTTS) (3,239, Modified MIT, TTS).
## Per-App Notes
- **OpenClaw** — self-hosted personal AI agent / orchestration layer (not a
model). Connects to cloud APIs, subscriptions, API providers, and **local
models**; drives files, browser, terminal/shell, and messaging channels. 2.0
rebuilt the browser app as the primary interface, added shared cloud sessions,
simplified onboarding. Node 24 (22.14+) + Docker; minimal footprint for
API-based LLMs (2 vCPU / 2–4 GB RAM). **Caveat:** community reaction to 2.0
was mixed — migration problems, broken gateways, lost automations, model-auth
issues (InfoQ). Tool/shell-access agent → strongest isolation.
- **Hermes Agent** — Nous Research's personal agent, centered on persistent
memory and cross-platform messaging. Model-agnostic; Docker/Podman; config in
`~/.hermes/config.yaml`. Releases weekly. Tool-access agent → isolate.
- **Ollama** — the de-facto standard local LLM runtime; OpenAI-compatible API
(default `:11434`) that nearly every app in this list consumes. The default
provider for JobSync, Resume Matcher, AnythingLLM, and more.
- **Dify** — full LLMOps/agent platform: prompt IDE, RAG pipeline,
agent/workflow builder, model management. `docker compose` full stack.
Modified Apache-2.0: commercial use allowed, but **multi-tenant operation
requires a commercial license** (one tenant = one workspace) — fine for a
single-workspace homelab.
- **Open WebUI** — the dominant ChatGPT-style chat UI: multi-user auth, model
switching, tools, pipelines, RAG, image gen (ComfyUI as engine), agents.
Single container; connect to Ollama / llama.cpp / vLLM via base-URL. Already
this fleet's chat frontend.
- **ComfyUI** — node-based image *and* video generation (SD/SDXL/Flux/LLaDA/
Qwen-Image). The most popular local image-gen frontend in 2026. Repo moved to
the `Comfy-Org` org. Already running on this fleet (Qwen-Image 2.1 GGUF) and
wired as the Open WebUI image engine.
- **llama.cpp** — the reference GGUF inference engine; `llama-server` exposes an
OpenAI-compatible endpoint. Backbone of this fleet's frank-zealot/frank-dragoon.
- **vLLM** — high-throughput PagedAttention serving with an OpenAI-compatible
API; the multi-GPU option (frank-medium runs vLLM tensor-parallel across two
R9700s).
- **RAGFlow** — RAG engine with deep document understanding (PDFs, spreadsheets,
scans) and source citations. `docker compose` full stack; runs fully against
local models (Ollama).
- **LobeChat** — polished chat UI grown into an agent-operations layer (plugins,
agents, scheduled tasks). LobeHub Community License: commercial use allowed,
but **distributing a derivative requires a commercial license**.
- **Unsloth** — the leading open-source fine-tuning library (LoRA/QLoRA/SFT) +
GGUF quantization + a local run/train UI. GPU required; a batch job, not a
long-running service.
- **AnythingLLM** — local-first RAG + agent workspace: document ingestion, chat
over your data, agent mode, multi-user. Ollama first-class.
- **whisper.cpp** — OpenAI Whisper in C/C++; lightweight fully-local STT on CPU /
Metal / CUDA; `whisper-server` for an HTTP API.
- **LibreChat** — multi-provider, multi-user chat UI with agents, MCP, actions.
`docker compose` (app + MongoDB + Redis).
- **Tabby** — self-hosted AI coding assistant (Copilot-style) against your own
models. Apache-2.0 core; `ee/` directory under a separate enterprise license.
Last push 2026-06-30 (quieter since).
## Homelab Fit (Podman/Quadlet + Caddy + local LLM endpoints)
| App | Fit | Notes for this fleet |
|-----|-----|----------------------|
| **Open WebUI** | **Excellent** | Single container; point at existing llama.cpp/vLLM/Ollama via base-URL. Already the fleet's chat frontend. |
| **ComfyUI** | **Excellent** | GPU container (already running for Qwen-Image 2.1 GGUF). Serves the Open WebUI image-gen path. |
| **Ollama** | **Excellent** | Easiest OpenAI-compatible runtime for the other apps; expose on LAN / `host.containers.internal`. |
| **Dify** | **Good** | compose → Quadlet. Multi-tenant restriction moot for single-workspace. Heavier stack (DB + Redis + vector store). |
| **RAGFlow** | **Good** | compose → Quadlet. Runs fully against local models. Budget disk for the document index. |
| **AnythingLLM** | **Good** | Single container, Ollama-first, low footprint. Good local RAG entry point. |
| **LibreChat** | **Good** | compose (app + Mongo + Redis) → Quadlet. Multi-user, MCP/agents. |
| **LobeChat** | **Good** | Single Postgres-backed container. Watch the Community License if redistributing a derivative. |
| **whisper.cpp / faster-whisper** | **Good** | Lightweight STT; `whisper-server` behind Caddy for a LAN transcription API. CPU-fine. |
| **Piper / Kokoro / Chatterbox** | **Good** | Local TTS; pair with whisper.cpp for a fully-local voice loop. Chatterbox wants a GPU; Piper/Kokoro run on CPU. |
| **Tabby** | **Good** | compose → Quadlet; self-hosted coding assistant against local models. |
| **Unsloth** | **OK (one-off)** | Fine-tuning is a batch job, not a service. Run transiently on a GPU server, then serve the GGUF via llama.cpp. |
| **OpenClaw / Hermes** | **Good, strong isolation** | Node.js / container agent pointing at local LLMs. Highest buzz, but tool/shell access — non-root, restricted network, vetted skills, behind the guard chain. |
**GPU co-tenancy note:** ComfyUI, vLLM, Unsloth, and Chatterbox all want VRAM.
Size new containers against the existing LLM endpoints' held VRAM to avoid
OOMing a production endpoint. The macOS mini is a natural home for the
low-footprint, CPU-friendly apps (Open WebUI, AnythingLLM, whisper.cpp,
Piper/Kokoro, LibreChat, LobeChat) via Metal-backed llama.cpp/Ollama.
## Best Practices: Adopting a New App Into a Guard-Chained Homelab
1. **Route everything through the existing guard chain.** Put the app behind
Caddy under the same Anubis+Authentik chain the fleet runs. Do not expose a
new port directly; use the published-port cutover gate (confirm the port is
free before starting, or pasta's bind fails and the failure hides behind
`Restart=on-failure`).
2. **Enable app-native auth and add SSO on top.** Turn on the app's built-in
multi-user auth *and* gate it with Authentik SSO at the proxy.
3. **Prefer local endpoints for the AI; keep PII on-prem.** Point RAG/chat/agent
apps at local llama.cpp/vLLM/Ollama (base-URL override) so prompts and
documents never leave the LAN. Reach the host runtime via
`host.containers.internal`, not `localhost`.
4. **Use Quadlet for boot persistence.** Convert every compose/`docker run` to
Quadlet units. Remember the anonymous-volume trap — use named volumes for
all app data.
5. **Pin image tags and track release notes.** The window's clearest lesson:
**OpenClaw 2.0 (2026-09-01) caused migration problems, broken gateways, and
lost automations for some users.** Pin tags, read release notes, stage
upgrades (test on the mini / a spare server first).
6. **Isolate agents aggressively.** OpenClaw and Hermes have tool/shell/browser
access. Non-root, restricted network, vetted skills/plugins only, full
logging of tool execution.
7. **Treat ingested content as untrusted (prompt injection).** Keep a human
review step before any action; scope agent tool access to the minimum.
8. **Protect the secrets the app stores.** API keys in env vars (or a vault),
never committed. Set the encryption key *before* adding keys, never change
it after.
9. **Budget VRAM and co-tenancy.** Measure a new GPU container's VRAM against
the endpoints already resident on the card.
10. **Back up the data volumes.** Snapshot the app's data volume; agents store
memory/state that is expensive to rebuild.
11. **Mind the license.** ComfyUI is GPL-3.0; Dify and LobeChat are
modified-Apache with multi-tenant / derivative restrictions; Tabby has an
`ee/` directory. Fine for a single-workspace, non-redistributed homelab, but
note them before derivative work.
# Citations
- OpenClaw: https://github.com/openclaw/openclaw (MIT, 390,235 stars)
- Hermes Agent: https://github.com/NousResearch/hermes-agent (MIT, 247,907 stars)
- Ollama: https://github.com/ollama/ollama (MIT, 181,444 stars)
- Dify: https://github.com/langgenius/dify (Modified Apache-2.0, 156,835 stars)
- Open WebUI: https://github.com/open-webui/open-webui (Open WebUI License, 152,784 stars)
- ComfyUI: https://github.com/Comfy-Org/ComfyUI (GPL-3.0, 134,407 stars)
- llama.cpp: https://github.com/ggml-org/llama.cpp (MIT, 129,149 stars)
- vLLM: https://github.com/vllm-project/vllm (Apache-2.0, 92,405 stars)
- RAGFlow: https://github.com/infiniflow/ragflow (Apache-2.0, 91,138 stars)
- LobeChat: https://github.com/lobehub/lobehub (LobeHub Community License, 82,748 stars)
- Unsloth: https://github.com/unslothai/unsloth (Apache-2.0, 76,560 stars)
- AnythingLLM: https://github.com/Mintplex-Labs/anything-llm (MIT, 66,318 stars)
- whisper.cpp: https://github.com/ggml-org/whisper.cpp (MIT, 53,852 stars)
- LibreChat: https://github.com/danny-avila/LibreChat (MIT, 44,608 stars)
- Tabby: https://github.com/TabbyML/tabby (Apache-2.0 + ee/, 33,889 stars)
- Chatterbox: https://github.com/resemble-ai/chatterbox (MIT, 26,515 stars)
- faster-whisper: https://github.com/SYSTRAN/faster-whisper (MIT, 25,512 stars)
- Piper TTS: https://github.com/rhasspy/piper (MIT, 11,287 stars)
- Kokoro-82M: https://github.com/hexgrad/kokoro (Apache-2.0, 8,933 stars)
- MisoTTS: https://github.com/MisoLabsAI/MisoTTS (Modified MIT, 3,239 stars)
- OpenClaw 2.0 release coverage: https://www.infoq.com/news/2026/09/openclaw-2-release/
- OpenClaw self-hosted architecture + security: https://hostkey.com/blog/208-what-is-openclaw-complete-self-hosted-ai-agent-guide/
- Release cadence: GitHub releases API, 2026-09-22
serverAppConfigs-docs/Research/SGLang.md
-154
View File
@@ -1,154 +0,0 @@
---
type: reference
title: SGLang
description: "What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet."
tags:
- sglang
- serving
- inference
- structured-output
- radix-attention
- llama.cpp
timestamp: 2026-09-01
---
# SGLang
SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
serving engine for large (and vision-language) models, from the LMSYS / UC
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
open-source serving engines, alongside vLLM. People talk about it because it
makes "call the model, branch, force the output into JSON" programs *fast*.
## What it is — in one paragraph
You serve an LLM by writing a small program: call the model, then based on its
output call it again, branch into parallel attempts, and force the answer into a
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
*see* the program's structure, then exploits it: it caches and reuses the model's
"working memory" (the KV cache) across requests that share a prompt prefix,
batches new requests into running generations without waiting for a batch to
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
illegal tokens one step at a time — or several at once. Reported payoff:
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
TGI, and similar.
## Core architecture
Two halves that can also run independently:
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
`extend`) for multi-call LLM programs with branching, parallel sampling, tool
calls, and structured output.
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
structured decoding work with a vanilla OpenAI client.
Building blocks:
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
tensors so the GPU sampler does one bulk copy per batch per step (kills the
CPU scheduler bottleneck).
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
non-contiguously in a global pool; mirrors vLLM's PagedAttention.
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
walks to the longest cached prefix, reuses those KV blocks, and only prefills
the divergent suffix. Payoff is compute + throughput for long shared prefixes
(RAG, agents), not raw memory.
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
graphs + torch.compile).
- **Structured / output-guided generation** — compiles the constraint (JSON
schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
disallowed logits; **jump-forward** decoding emits forced token runs in one step.
Backends: XGrammar (default), Outlines.
## SGLang vs llama.cpp vs vLLM
| Dimension | SGLang | llama.cpp |
|-----------|--------|-----------|
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
| Structured output | first-class (XGrammar, jump-forward) | narrower |
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
| Multi-GPU | native tensor parallelism | layer-split, more manual |
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
| License | Apache 2.0 | MIT |
**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
the gap narrows vs modern vLLM — benchmark on your own traffic.
## Deployment model & gotchas
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
tree). Size with `--mem-fraction-static`, `--context-length`,
`--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
`pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
`petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
**datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
(gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
degenerates). Single-replica trees need prefix-aware routing across a fleet
(`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
## Relevance to our servers
### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
for KV. MoE expert parallelism is supported.
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
HauhauCS checkpoint on Hugging Face.
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
sequential CPU pipeline. Simple low-concurrency chat → the operational weight
may not pay off.
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
SGLang-on-ROCm is supported (Docker/pip), but:
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
image as a proof-of-concept. Treat it as an enhancement/experiment, not a
production swap, until a quant for the MTP model exists and perf is measured on
gfx1151.
## Tradeoffs at a glance
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
trio for a CPU-only box).
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
## See Also
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
- [[Quantization by Hardware Architecture]]
## Citations
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
serverAppConfigs-docs/Research/index.md
+6 -7
View File
@@ -1,22 +1,21 @@
# Research
Subagent research briefs for the two hosts. Each page is a focused research
findings doc; the project root index ([serverAppConfigs-docs/index.md](../index.md)) links back.
findings doc; the project root index () links back.
## LLM Inference
- [[Llama Cpp Runtimes]] — The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine.
- [[Llama Cpp Code Diff]] — Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards.
- [[SGLang]] — What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet.
- The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine.
- Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards.
- What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet.
## Image / Media
- [[ComfyUI ROCm and Vulkan]] — Whether ComfyUI can use the Strix Halo APU at all, and which backend path actually reaches the GPU.
- [[Lemonade and sd.cpp]] — Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot.
## Audio / TTS
- [[Local TTS and Voice Cloning]] — Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack.
- Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack.
## Career / Job Search
@@ -24,7 +23,7 @@ findings doc; the project root index ([serverAppConfigs-docs/index.md](../index.
## Self-Hosted Landscape
- [[Most Popular Self-Hosted AI Apps]] — The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes.
- The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes.
## Games / Creative