dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

Browse Source 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
This commit is contained in:
Frank committed 2026-09-22 02:09:41 -07:00
1 parent 24f7a7772d
commit 93d1181fee
103 files changed
+2047 -4329
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
+240
View File
@@ -0,0 +1,240 @@
---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
tags:
- research
- tts
- voice-cloning
- audio
- stt
- vllm-omni
- kokoro
timestamp: 2026-09-22T00:00:00Z
---
# Local TTS and Voice Cloning
Research brief (2026-09-22) on creating a new local TTS voice from sample
files, the open-weight voice-cloning landscape, the full local AI audio
stack, and a recommended deployment for the fleet. Every claim is cited in
[Citations](#citations). The fleet has no audio input today (that capability
left with the retired NPU endpoint), so this is greenfield.
## Summary
- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
Voices are a fixed pack. A community fine-tune recipe exists
(`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
the output is Kokoro-styled, and OMP's local backend only accepts its
built-in voice list (a custom pack needs a patch).
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
(Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT)**, **VoxCPM2
(Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration — definitive:** OMP's `tts` tool supports exactly three
backends (`providers.tts` = `auto|local|xai|deepinfra`, verified in the
installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud.
**There is no generic OpenAI-compatible TTS endpoint in OMP today.** Two real
paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only)
— works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend
pointing at a local `vLLM-Omni` server.
- **Recommended stack:** `vllm/vllm-omni` serving **Qwen3-TTS-1.7B-Base** on the
3090 (OpenAI-compatible `POST /v1/audio/speech` + `POST /v1/audio/voices`
voice upload), **Parakeet TDT 1.1B** for English STT, **Silero VAD** embedded,
**Chatterbox** as the ultra-light English-only alternative. All fit in <8 GB
VRAM and co-locate with the existing LLM services.
## Can Kokoro learn a new voice?
Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
params, non-autoregressive) ships **fixed voice packs** — there is no
reference-audio path, so zero-shot cloning is not an option. OMP's local
backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
from a built-in list (`tts.localVoice`).
The best-documented fine-tune path is
[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):
| Stage | Data | Purpose |
|---|---|---|
| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
| Extraction | — | voice-pack (.pt) extraction for inference |
**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
and patch OMP to load the custom pack. The effort/quality trade is worse than
running a modern zero-shot cloner on the same GPU.
## 2026 voice-cloning landscape (permissive-licensed, single-GPU)
Ranked for a 3090 homelab, English-primary, personal voice:
1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference;
110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). *Top pick for a personal
English voice.* No official server (library + Gradio; wrap in FastAPI).
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
vLLM-Omni serving with `POST /v1/audio/voices` upload (closest to "named
voice" semantics), official fine-tuning code. *Top pick for a server-grade,
OpenAI-compatible deployment.*
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
bi-streaming, FastAPI/WebSocket server.
4. **GLM-TTS (MIT)** — best published CER (0.89), RL emotion control, zh/en.
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
Docker images.
Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
encoder not released); Voxtral TTS (no official cloning — codec encoder not
released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
(code withdrawn 2025-09).
## OMP integration (definitive)
OMP's `tts` tool (hidden by default; enable with `/tts`) supports exactly three
backends, selected by `providers.tts`:
| Backend | Endpoint (hardcoded in binary) | Voices |
|---|---|---|
| `local` | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` | built-in list (`tts.localVoice`) |
| `xai` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + any custom voice ID |
| `deepinfra` | `POST https://api.deepinfra.com/v1/openai/audio/speech` | `voice_id` when set |
There is **no generic OpenAI-compatible TTS endpoint** — `deepinfra` is a
hardcoded URL, not a configurable base URL, and OMP's `models.yml` custom
providers (LLM chat only) do not wire into the `tts` tool.
Paths for a Frank-cloned voice:
1. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → `tts` tool.
Free (≤30 voices), cloud, US-only.
2. **Local, full stack — small OMP patch.** OMP is open source; the DeepInfra
backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A
~50-line `openai` backend (configurable `base_url` + `voice`) lets OMP call
any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
3. **No-patch workaround.** OMP's `bash` tool can `curl` the local TTS server
ad hoc.
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.
## Full local AI audio stack
**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
60-min, 50+ langs).
**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
faster-whisper/whisperX).
**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
official Docker `vllm/vllm-omni` CUDA and `vllm/vllm-omni-rocm` ROCm, e.g.
`:v0.28.0`) is the one server exposing the standard OpenAI speech API
(`POST /v1/audio/speech` + `POST /v1/audio/voices` upload for Qwen3-TTS) across
the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on
gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others:
Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS
Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own
format, not OpenAI-shaped).
**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
cloning; the Perth watermark is built into Chatterbox.
## Recommended deployment (Dankclaw)
TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it
all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the
Dankserver R9700s stay dedicated to frank-medium.
| Service | Image / install | Port | GPU | Notes |
|---|---|---|---|---|
| TTS (primary, cloning) | `vllm/vllm-omni:v0.28.0` · `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35` | 8881 | 3090 (~6 GB) | OpenAI `/v1/audio/speech`; upload Frank's voice once via `/v1/audio/voices`; 3 s cloning; 10 langs; Apache-2.0 |
| TTS (light, English) | Chatterbox (MIT) in a small FastAPI container | 8880 | 3090 (~1 GB) / CPU | 10 s ref clip; wrap `chatterbox.generate` in ~30 lines |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | CPU | CC-BY-4.0; WER 1.39% |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | CPU / iGPU | MIT |
| VAD | Silero (embedded) | — | CPU | MIT |
Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
## Voice-creation workflow (the "from sample files" answer)
1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
GLM-TTS clone from it per request (no training).
3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
(vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
backend — highest effort, Kokoro-styled output.
## Best practices
- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
fine-tune on quality and effort.
- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
clean voice with Demucs if the source is music/mixed.
- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
for many TTS models — it is the only server with the standard
`/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
- **Check the license before serving publicly:** several strong cloners are
non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
TTS engine shares the card with frank-small; move to the R9700s only if TTS
throughput demand grows.
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
internal address, never the public hostname.
## Citations
- OMP TTS tool: https://omp.sh/docs/tools/tts · installed binary
`@oh-my-pi/pi-coding-agent/dist/cli.js` (endpoints, voices, model IDs)
- Kokoro: https://github.com/hexgrad/Kokoro ·
https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
- Chatterbox: https://github.com/resemble-ai/chatterbox
- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
https://huggingface.co/zai-org/GLM-TTS
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
https://huggingface.co/openbmb/VoxCPM2
- F5-TTS: https://github.com/SWivid/F5-TTS
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
- IndexTTS: https://github.com/index-tts/index-tts
- Fish Speech: https://github.com/fishaudio/fish-speech
- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
- XTTS: https://github.com/coqui-ai/TTS
- MegaTTS3: https://github.com/bytedance/MegaTTS3
- Higgs Audio: https://github.com/boson-ai/higgs-audio
- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
- Parler-TTS: https://github.com/huggingface/parler-tts
- VibeVoice: https://github.com/microsoft/VibeVoice
- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
https://docs.vllm.ai/projects/vllm-omni/en/latest/
- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
whisperX: https://github.com/m-bain/whisperX
- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
Canary: https://huggingface.co/nvidia/canary-1b-v2
- Moonshine: https://github.com/moonshine-ai/moonshine
- Silero VAD: https://github.com/snakers4/silero-vad
- xAI custom voices:
https://docs.x.ai/developers/model-capabilities/audio/custom-voices