serverAppConfigs/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md

---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
tags:
  - research
  - tts
  - voice-cloning
  - audio
  - stt
  - vllm-omni
  - kokoro
timestamp: 2026-09-22T00:00:00Z
---

# Local TTS and Voice Cloning

Research brief (2026-09-22) on creating a new local TTS voice from sample
files, the open-weight voice-cloning landscape, the full local AI audio
stack, and a recommended deployment for the fleet. Every claim is cited in
[Citations](#citations). The fleet has no audio input today (that capability
left with the retired NPU endpoint), so this is greenfield.

## Summary

- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
  Voices are a fixed pack. A community fine-tune recipe exists
  (`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
  single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
  the output is Kokoro-styled, and OMP's local backend only accepts its
  built-in voice list (a custom pack needs a patch).
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
  leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
  (Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT)**, **VoxCPM2
  (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration — definitive:** OMP's `tts` tool supports exactly three
  backends (`providers.tts` = `auto|local|xai|deepinfra`, verified in the
  installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud.
  **There is no generic OpenAI-compatible TTS endpoint in OMP today.** Two real
  paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only)
  — works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend
  pointing at a local `vLLM-Omni` server.
- **Recommended stack:** `vllm/vllm-omni` serving **Qwen3-TTS-1.7B-Base** on the
  3090 (OpenAI-compatible `POST /v1/audio/speech` + `POST /v1/audio/voices`
  voice upload), **Parakeet TDT 1.1B** for English STT, **Silero VAD** embedded,
  **Chatterbox** as the ultra-light English-only alternative. All fit in <8 GB
  VRAM and co-locate with the existing LLM services.

## Can Kokoro learn a new voice?

Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
params, non-autoregressive) ships **fixed voice packs** — there is no
reference-audio path, so zero-shot cloning is not an option. OMP's local
backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
from a built-in list (`tts.localVoice`).

The best-documented fine-tune path is
[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):

| Stage | Data | Purpose |
|---|---|---|
| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
| Extraction | — | voice-pack (.pt) extraction for inference |

**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
and patch OMP to load the custom pack. The effort/quality trade is worse than
running a modern zero-shot cloner on the same GPU.

## 2026 voice-cloning landscape (permissive-licensed, single-GPU)

Ranked for a 3090 homelab, English-primary, personal voice:

1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference;
   110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). *Top pick for a personal
   English voice.* No official server (library + Gradio; wrap in FastAPI).
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
   vLLM-Omni serving with `POST /v1/audio/voices` upload (closest to "named
   voice" semantics), official fine-tuning code. *Top pick for a server-grade,
   OpenAI-compatible deployment.*
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
   bi-streaming, FastAPI/WebSocket server.
4. **GLM-TTS (MIT)** — best published CER (0.89), RL emotion control, zh/en.
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
   of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
   Docker images.

Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
encoder not released); Voxtral TTS (no official cloning — codec encoder not
released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
(code withdrawn 2025-09).

## OMP integration (definitive)

OMP's `tts` tool (hidden by default; enable with `/tts`) supports exactly three
backends, selected by `providers.tts`:

| Backend | Endpoint (hardcoded in binary) | Voices |
|---|---|---|
| `local` | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` | built-in list (`tts.localVoice`) |
| `xai` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + any custom voice ID |
| `deepinfra` | `POST https://api.deepinfra.com/v1/openai/audio/speech` | `voice_id` when set |

There is **no generic OpenAI-compatible TTS endpoint** — `deepinfra` is a
hardcoded URL, not a configurable base URL, and OMP's `models.yml` custom
providers (LLM chat only) do not wire into the `tts` tool.

Paths for a Frank-cloned voice:

1. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
   speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → `tts` tool.
   Free (≤30 voices), cloud, US-only.
2. **Local, full stack — small OMP patch.** OMP is open source; the DeepInfra
   backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A
   ~50-line `openai` backend (configurable `base_url` + `voice`) lets OMP call
   any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
3. **No-patch workaround.** OMP's `bash` tool can `curl` the local TTS server
   ad hoc.
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.

## Full local AI audio stack

**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
60-min, 50+ langs).

**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
faster-whisper/whisperX).

**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
official Docker `vllm/vllm-omni` CUDA and `vllm/vllm-omni-rocm` ROCm, e.g.
`:v0.28.0`) is the one server exposing the standard OpenAI speech API
(`POST /v1/audio/speech` + `POST /v1/audio/voices` upload for Qwen3-TTS) across
the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on
gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others:
Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS
Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own
format, not OpenAI-shaped).

**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
cloning; the Perth watermark is built into Chatterbox.

## Recommended deployment (Dankclaw)

TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it
all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the
Dankserver R9700s stay dedicated to frank-medium.

| Service | Image / install | Port | GPU | Notes |
|---|---|---|---|---|
| TTS (primary, cloning) | `vllm/vllm-omni:v0.28.0` · `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35` | 8881 | 3090 (~6 GB) | OpenAI `/v1/audio/speech`; upload Frank's voice once via `/v1/audio/voices`; 3 s cloning; 10 langs; Apache-2.0 |
| TTS (light, English) | Chatterbox (MIT) in a small FastAPI container | 8880 | 3090 (~1 GB) / CPU | 10 s ref clip; wrap `chatterbox.generate` in ~30 lines |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | CPU | CC-BY-4.0; WER 1.39% |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | CPU / iGPU | MIT |
| VAD | Silero (embedded) | — | CPU | MIT |

Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).

## Voice-creation workflow (the "from sample files" answer)

1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
   GLM-TTS clone from it per request (no training).
3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
   (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
   or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
   backend — highest effort, Kokoro-styled output.

## Best practices

- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
  cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
  fine-tune on quality and effort.
- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
  clean voice with Demucs if the source is music/mixed.
- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
  for many TTS models — it is the only server with the standard
  `/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
- **Check the license before serving publicly:** several strong cloners are
  non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
  licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
  Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
- **Co-locate on the 3090** with a low `--gpu-memory-utilization` (~0.35) so the
  TTS engine shares the card with frank-small; move to the R9700s only if TTS
  throughput demand grows.
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
  Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
  internal address, never the public hostname.

## Citations

- OMP TTS tool: https://omp.sh/docs/tools/tts · installed binary
  `@oh-my-pi/pi-coding-agent/dist/cli.js` (endpoints, voices, model IDs)
- Kokoro: https://github.com/hexgrad/Kokoro ·
  https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
- Chatterbox: https://github.com/resemble-ai/chatterbox
- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
  https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
  https://huggingface.co/zai-org/GLM-TTS
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
  https://huggingface.co/openbmb/VoxCPM2
- F5-TTS: https://github.com/SWivid/F5-TTS
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
- IndexTTS: https://github.com/index-tts/index-tts
- Fish Speech: https://github.com/fishaudio/fish-speech
- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
- XTTS: https://github.com/coqui-ai/TTS
- MegaTTS3: https://github.com/bytedance/MegaTTS3
- Higgs Audio: https://github.com/boson-ai/higgs-audio
- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
- Parler-TTS: https://github.com/huggingface/parler-tts
- VibeVoice: https://github.com/microsoft/VibeVoice
- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
  https://docs.vllm.ai/projects/vllm-omni/en/latest/
- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
  faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
  whisperX: https://github.com/m-bain/whisperX
- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
  Canary: https://huggingface.co/nvidia/canary-1b-v2
- Moonshine: https://github.com/moonshine-ai/moonshine
- Silero VAD: https://github.com/snakers4/silero-vad
- xAI custom voices:
  https://docs.x.ai/developers/model-capabilities/audio/custom-voices