Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
241 lines
13 KiB
Markdown
241 lines
13 KiB
Markdown
---
|
||
type: research
|
||
title: Local TTS and Voice Cloning
|
||
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
|
||
tags:
|
||
- research
|
||
- tts
|
||
- voice-cloning
|
||
- audio
|
||
- stt
|
||
- vllm-omni
|
||
- kokoro
|
||
timestamp: 2026-09-22T00:00:00Z
|
||
---
|
||
|
||
# Local TTS and Voice Cloning
|
||
|
||
Research brief (2026-09-22) on creating a new local TTS voice from sample
|
||
files, the open-weight voice-cloning landscape, the full local AI audio
|
||
stack, and a recommended deployment for the fleet. Every claim is cited in
|
||
[Citations](#citations). The fleet has no audio input today (that capability
|
||
left with the retired NPU endpoint), so this is greenfield.
|
||
|
||
## Summary
|
||
|
||
- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
|
||
Voices are a fixed pack. A community fine-tune recipe exists
|
||
(`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
|
||
single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
|
||
the output is Kokoro-styled, and OMP's local backend only accepts its
|
||
built-in voice list (a custom pack needs a patch).
|
||
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
|
||
leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
|
||
(Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT)**, **VoxCPM2
|
||
(Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
|
||
- **OMP integration — definitive:** OMP's `tts` tool supports exactly three
|
||
backends (`providers.tts` = `auto|local|xai|deepinfra`, verified in the
|
||
installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud.
|
||
**There is no generic OpenAI-compatible TTS endpoint in OMP today.** Two real
|
||
paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only)
|
||
— works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend
|
||
pointing at a local `vLLM-Omni` server.
|
||
- **Recommended stack:** `vllm/vllm-omni` serving **Qwen3-TTS-1.7B-Base** on the
|
||
3090 (OpenAI-compatible `POST /v1/audio/speech` + `POST /v1/audio/voices`
|
||
voice upload), **Parakeet TDT 1.1B** for English STT, **Silero VAD** embedded,
|
||
**Chatterbox** as the ultra-light English-only alternative. All fit in <8 GB
|
||
VRAM and co-locate with the existing LLM services.
|
||
|
||
## Can Kokoro learn a new voice?
|
||
|
||
Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
|
||
params, non-autoregressive) ships **fixed voice packs** — there is no
|
||
reference-audio path, so zero-shot cloning is not an option. OMP's local
|
||
backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
|
||
from a built-in list (`tts.localVoice`).
|
||
|
||
The best-documented fine-tune path is
|
||
[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):
|
||
|
||
| Stage | Data | Purpose |
|
||
|---|---|---|
|
||
| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
|
||
| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
|
||
| Extraction | — | voice-pack (.pt) extraction for inference |
|
||
|
||
**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
|
||
you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
|
||
and patch OMP to load the custom pack. The effort/quality trade is worse than
|
||
running a modern zero-shot cloner on the same GPU.
|
||
|
||
## 2026 voice-cloning landscape (permissive-licensed, single-GPU)
|
||
|
||
Ranked for a 3090 homelab, English-primary, personal voice:
|
||
|
||
1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference;
|
||
110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). *Top pick for a personal
|
||
English voice.* No official server (library + Gradio; wrap in FastAPI).
|
||
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
|
||
vLLM-Omni serving with `POST /v1/audio/voices` upload (closest to "named
|
||
voice" semantics), official fine-tuning code. *Top pick for a server-grade,
|
||
OpenAI-compatible deployment.*
|
||
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
|
||
bi-streaming, FastAPI/WebSocket server.
|
||
4. **GLM-TTS (MIT)** — best published CER (0.89), RL emotion control, zh/en.
|
||
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
|
||
of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
|
||
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
|
||
7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
|
||
Docker images.
|
||
|
||
Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
|
||
(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
|
||
license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
|
||
encoder not released); Voxtral TTS (no official cloning — codec encoder not
|
||
released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
|
||
(code withdrawn 2025-09).
|
||
|
||
## OMP integration (definitive)
|
||
|
||
OMP's `tts` tool (hidden by default; enable with `/tts`) supports exactly three
|
||
backends, selected by `providers.tts`:
|
||
|
||
| Backend | Endpoint (hardcoded in binary) | Voices |
|
||
|---|---|---|
|
||
| `local` | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` | built-in list (`tts.localVoice`) |
|
||
| `xai` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + any custom voice ID |
|
||
| `deepinfra` | `POST https://api.deepinfra.com/v1/openai/audio/speech` | `voice_id` when set |
|
||
|
||
There is **no generic OpenAI-compatible TTS endpoint** — `deepinfra` is a
|
||
hardcoded URL, not a configurable base URL, and OMP's `models.yml` custom
|
||
providers (LLM chat only) do not wire into the `tts` tool.
|
||
|
||
Paths for a Frank-cloned voice:
|
||
|
||
1. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
|
||
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → `tts` tool.
|
||
Free (≤30 voices), cloud, US-only.
|
||
2. **Local, full stack — small OMP patch.** OMP is open source; the DeepInfra
|
||
backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A
|
||
~50-line `openai` backend (configurable `base_url` + `voice`) lets OMP call
|
||
any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
|
||
3. **No-patch workaround.** OMP's `bash` tool can `curl` the local TTS server
|
||
ad hoc.
|
||
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.
|
||
|
||
## Full local AI audio stack
|
||
|
||
**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
|
||
VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
|
||
timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
|
||
LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
|
||
Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
|
||
60-min, 50+ langs).
|
||
|
||
**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
|
||
faster-whisper/whisperX).
|
||
|
||
**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
|
||
official Docker `vllm/vllm-omni` CUDA and `vllm/vllm-omni-rocm` ROCm, e.g.
|
||
`:v0.28.0`) is the one server exposing the standard OpenAI speech API
|
||
(`POST /v1/audio/speech` + `POST /v1/audio/voices` upload for Qwen3-TTS) across
|
||
the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on
|
||
gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others:
|
||
Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS
|
||
Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own
|
||
format, not OpenAI-shaped).
|
||
|
||
**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
|
||
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
|
||
cloning; the Perth watermark is built into Chatterbox.
|
||
|
||
## Recommended deployment (Dankclaw)
|
||
|
||
TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it
|
||
all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the
|
||
Dankserver R9700s stay dedicated to frank-medium.
|
||
|
||
| Service | Image / install | Port | GPU | Notes |
|
||
|---|---|---|---|---|
|
||
| TTS (primary, cloning) | `vllm/vllm-omni:v0.28.0` · `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35` | 8881 | 3090 (~6 GB) | OpenAI `/v1/audio/speech`; upload Frank's voice once via `/v1/audio/voices`; 3 s cloning; 10 langs; Apache-2.0 |
|
||
| TTS (light, English) | Chatterbox (MIT) in a small FastAPI container | 8880 | 3090 (~1 GB) / CPU | 10 s ref clip; wrap `chatterbox.generate` in ~30 lines |
|
||
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | CPU | CC-BY-4.0; WER 1.39% |
|
||
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | CPU / iGPU | MIT |
|
||
| VAD | Silero (embedded) | — | CPU | MIT |
|
||
|
||
Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).
|
||
|
||
## Voice-creation workflow (the "from sample files" answer)
|
||
|
||
1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
|
||
2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
|
||
GLM-TTS clone from it per request (no training).
|
||
3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
|
||
(vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
|
||
or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
|
||
4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
|
||
backend — highest effort, Kokoro-styled output.
|
||
|
||
## Best practices
|
||
|
||
- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
|
||
cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
|
||
fine-tune on quality and effort.
|
||
- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
|
||
clean voice with Demucs if the source is music/mixed.
|
||
- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
|
||
for many TTS models — it is the only server with the standard
|
||
`/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
|
||
- **Check the license before serving publicly:** several strong cloners are
|
||
non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
|
||
licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
|
||
Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
|
||
- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
|
||
TTS engine shares the card with frank-zealot; move to the R9700s only if TTS
|
||
throughput demand grows.
|
||
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
|
||
Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
|
||
internal address, never the public hostname.
|
||
|
||
## Citations
|
||
|
||
- OMP TTS tool: https://omp.sh/docs/tools/tts · installed binary
|
||
`@oh-my-pi/pi-coding-agent/dist/cli.js` (endpoints, voices, model IDs)
|
||
- Kokoro: https://github.com/hexgrad/Kokoro ·
|
||
https://huggingface.co/spaces/hexgrad/Kokoro-TTS
|
||
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
|
||
- Chatterbox: https://github.com/resemble-ai/chatterbox
|
||
- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
|
||
https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
|
||
- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
|
||
- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
|
||
https://huggingface.co/zai-org/GLM-TTS
|
||
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
|
||
https://huggingface.co/openbmb/VoxCPM2
|
||
- F5-TTS: https://github.com/SWivid/F5-TTS
|
||
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
|
||
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
|
||
- IndexTTS: https://github.com/index-tts/index-tts
|
||
- Fish Speech: https://github.com/fishaudio/fish-speech
|
||
- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
|
||
- XTTS: https://github.com/coqui-ai/TTS
|
||
- MegaTTS3: https://github.com/bytedance/MegaTTS3
|
||
- Higgs Audio: https://github.com/boson-ai/higgs-audio
|
||
- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
|
||
- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
|
||
- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
|
||
- Parler-TTS: https://github.com/huggingface/parler-tts
|
||
- VibeVoice: https://github.com/microsoft/VibeVoice
|
||
- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
|
||
https://docs.vllm.ai/projects/vllm-omni/en/latest/
|
||
- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
|
||
- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
|
||
faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
|
||
whisperX: https://github.com/m-bain/whisperX
|
||
- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
|
||
Canary: https://huggingface.co/nvidia/canary-1b-v2
|
||
- Moonshine: https://github.com/moonshine-ai/moonshine
|
||
- Silero VAD: https://github.com/snakers4/silero-vad
|
||
- xAI custom voices:
|
||
https://docs.x.ai/developers/model-capabilities/audio/custom-voices
|