serverAppConfigs/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md

---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
tags:
  - research
  - tts
  - voice-cloning
  - audio
  - stt
  - vllm-omni
  - kokoro
timestamp: 2026-10-18T00:00:00Z
---

# Local TTS and Voice Cloning

Research brief (2026-09-22) on creating a new local TTS voice from sample
files, the open-weight voice-cloning landscape, the full local AI audio
stack, and a recommended deployment for the fleet. Every claim is cited in
[Citations](#citations). The fleet has no audio input today (that capability
left with the retired NPU endpoint, returned with the Gemma4 audio projector on
2026-09-17, and left again with that retirement on 2026-10-01 — audio is still
greenfield: every fleet endpoint declares `input: [text, image]`
(`~/.omp/agent/models.yml:63,149,214`).

## Summary

- **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.**
  Voices are a fixed pack. A community fine-tune recipe exists
  (`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred
  single-speaker samples, then voice-pack extraction. Feasible on one GPU, but
  the output is Kokoro-styled, and OMP's local backend only accepts its
  built-in voice list (a custom pack needs a patch).
- **The 2026 open-weight cloning landscape is strong.** Permissive-licensed
  leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS
  (Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT weights)**,
  **VoxCPM2 (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**.
- **OMP integration (re-verified 2026-10-18):** the speech tool is gated by
  `speechgen.enabled` and resolves models through the **`speech` role and its
  fallback chain** (`modelRoles.speech`; this workstation runs
  `local/kokoro`). Three APIs: local Kokoro ONNX (always WAV/PCM16), `xai-tts`,
  and **`openai-speech` — a generic OpenAI-compatible `/v1/audio/speech`
  client** (MP3 or WAV, `voice_id` forwarded only when explicitly set, 15 000
  characters max; DeepInfra is now just a model on that API). The 2026-09-22
  finding that OMP has *no* configurable OpenAI TTS endpoint is **retired**: a
  local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
  #2 rules out every GPU option (the 3090 has carried frank-zealot since
  2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF
  engine on **8880** behind a voice-registry shim on **8890**
  (`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI
  `/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on
  2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or
  8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality
  number has been measured for this shape**, so this page states none.

## Can Kokoro learn a new voice?

Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M
params, non-autoregressive) ships **fixed voice packs** — there is no
reference-audio path, so zero-shot cloning is not an option. OMP's local
backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice
from a built-in list (`tts.localVoice`).

The best-documented fine-tune path is
[`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0):

| Stage | Data | Purpose |
|---|---|---|
| Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune |
| Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization |
| Extraction | — | voice-pack (.pt) extraction for inference |

**Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but
you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe,
and patch OMP to load the custom pack. The effort/quality trade is worse than
running a modern zero-shot cloner on the same GPU.

## 2026 voice-cloning landscape (permissive-licensed, single-GPU)

Ranked (2026-09-22) for a 3090 homelab, English-primary, personal voice; the
2026-10-18 fleet decision below moved this work off the GPU entirely:

1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference
   (the README's own example is `your_10s_ref_clip.wav`); 110 M (Nano) to 0.5 B
   (Turbo/Multilingual); ~1–2 GB VRAM *(unverified 2026-10-18)*; Nano is
   documented for CPU — "3x faster than realtime on 8 CPU cores". *Top pick for
   a personal English voice.* No server in the README (library use only; wrap
   it yourself); every generated file carries Resemble's PerTh watermark.
2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming,
   vLLM-Omni serving with a `POST /v1/audio/voices` upload *(unverified
   2026-10-18 — the reachable docs do not list it)*, the closest thing to
   "named voice" semantics, official fine-tuning code. *Top pick for a
   OpenAI-compatible deployment.*
3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms
   bi-streaming, FastAPI and gRPC servers.
4. **GLM-TTS (MIT model weights, Apache-2.0 code)** — CER 0.89 after
   multi-reward GRPO (down from 1.03), RL emotion control, zh/en; the prompt
   audio shipped in the repo is marked research-only.
5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min
   of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.*
6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton.
7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI,
   Docker images.

Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS
(non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research
license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE
encoder not released); Voxtral TTS (no official cloning — codec encoder not
released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS
(code withdrawn 2025-09).

## OMP integration (definitive)

The tool ships hidden and is enabled by `speechgen.enabled`
(`~/.omp/agent/config.yml:222-223`); the harness docs list it as "Generate
speech". Models come from the **`speech` role and its fallback chain**, not
from a hardcoded backend list (`modelRoles.speech`,
`~/.omp/agent/config.yml:16` = `local/kokoro` today):

| API | Endpoint | Notes (from the installed bundle, 2026-10-18) |
|---|---|---|
| local-inference | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` (Apache-2.0) | no network, **always WAV/PCM16**, voice from the built-in list (`tts.localVoice`) |
| `xai-tts` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + custom voice IDs; MP3 or WAV |
| `openai-speech` | `POST <provider baseUrl>/audio/speech` | the generic OpenAI-compatible client — MP3 or WAV, `voice_id` forwarded **only when explicitly set**, 15 000-character cap on `input`; DeepInfra is one model entry on it (`https://api.deepinfra.com/v1/openai`) |

The 2026-09-22 finding — "**no generic OpenAI-compatible TTS endpoint** …
`deepinfra` is a hardcoded URL, not a configurable base URL" — is **retired**:
`openai-speech` is that configurable client, selected by putting a model on the
`speech` role. Still true from the original audit: the local Kokoro backend
accepts only its built-in voice list, so a custom pack needs a patch.

Paths for a Frank-cloned voice (2026-10-18):

1. **Local server + speech role, no patch.** Register the local engine (or the
   8890 voice shim) as a provider model on the `openai-speech` API and point
   `modelRoles.speech` at it; the shim turns a voice *name* into a reference
   clip and forwards it as base64 `reference_audio`
   (`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
   speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
   tool. Free (≤30 voices), cloud, US-only.
3. **No-patch workaround.** The `bash` tool can `curl` the local TTS server ad
   hoc, and the harness `tts` device writes a speech file to a path you give it.
4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled.

## Full local AI audio stack

**STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in
VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word
timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0,
LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 /
Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT,
60-min, 50+ langs).

**VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in
faster-whisper/whisperX).

**TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni),
Apache-2.0) publishes the official images `vllm/vllm-omni` (CUDA) and
`vllm/vllm-omni-rocm` (ROCm), both tagged `v0.28.0` (Docker Hub tag lists,
2026-10-18), and serves Qwen3-TTS (`docs/models/supported_models.md`). It
serves the OpenAI speech path `POST /v1/audio/speech`; the
`POST /v1/audio/voices` upload route is *(unverified 2026-10-18)* — the docs
reachable today do not document it. Its gfx942/gfx950 validation note is a
footnote scoped to MiniMax H3 diffusion, **not** to TTS, and the fleet runs no
vLLM anywhere: the R9700s carry radiance as frank-archon
(`scripts/serve-frank-archon.sh:2`). It is also no longer "the one server" with
the OpenAI shape — llama.cpp's own `llama-tts-server` has it too (next
section). Others: Kokoro-FastAPI (Docker, :8880, Apache-2.0), CosyVoice
**FastAPI and gRPC** (`runtime/python/fastapi`, `runtime/python/grpc`; the
README documents no WebSocket server), F5-TTS Gradio/Triton, GPT-SoVITS
WebUI/api_v2, Fish S2 `tools/api_server.py` (own format, not OpenAI-shaped).

**Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the
reference-clip standard); Demucs (`htdemucs`) for vocal separation before
cloning; the Perth watermark is built into Chatterbox.

## llama.cpp-native TTS serving (verified 2026-10-18)

The CPU shape does not need a Python server at all:

- **`llama-tts-server`** ([`tc-mb/llama.cpp-omni`](https://github.com/tc-mb/llama.cpp-omni),
  MIT, "Omni inference in C/C++") is a standalone OpenAI-compatible TTS server
  that runs without an LLM model (build target `llama-tts-server`).
  `POST /v1/audio/speech` takes `model` (`"voxcpm"`, also accepts `"voxcpm2"`),
  `input`, `voice` — documented **"reserved"**, so do not build named-voice
  semantics on it — `response_format` (`wav`/`pcm`), `reference_audio`
  (base64 WAV for cloning), `seed` 42, `cfg_value` 2.0, `inference_timesteps`
  10, `max_steps` 200, `temperature` 1.0. `POST /v1/audio/speech/stream`
  returns chunked WAV; `POST /v1/voxcpm2/init` hot-loads a runtime from
  `base_lm` + `acoustic` GGUFs with `n_gpu_layers` (`-1` = all);
  `GET /v1/audio/speech/models` lists what is loaded
  (`tools/omni/voxcpm2/README.md`:232, 321-354).
- **Mainline `llama-tts`** (ggml-org/llama.cpp) has `--tts-speaker-file` for a
  speaker reference with **Qwen3-TTS** and with **Kyutai Pocket-TTS** — the
  latter requires it ("the model produces almost no audio without it"), and
  Pocket-TTS converts one `languages/<name>` directory, never the repo root
  (`tools/tts/README.md`:20-58; `kyutai/pocket-tts` is CC-BY-4.0).
- **VoxCPM2 GGUF provenance:** `openbmb/VoxCPM2` publishes **PyTorch weights
  only** (`model.safetensors`, no `.gguf` sibling — HF API 2026-10-18). The
  GGUFs the fork loads come from
  [`DennisHuang648/VoxCPM2-GGUF`](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF)
  (`VoxCPM2-BaseLM-F16.gguf`, `VoxCPM2-BaseLM-Q8_0.gguf`,
  `VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
  `tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.

Nothing has been measured on this fleet for any of these engines — no RTF, no
latency, no MOS. That absence is the finding, not an omission.

## Recommended deployment (Dankclaw)

As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus a thin
voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl
--user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or
8890.

| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot |
| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |

8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM=
http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT
keep clear of the fleet's LLM ports 7770/7771.

## Voice-creation workflow (the "from sample files" answer)

1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/
   GLM-TTS clone from it per request (no training).
3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices`
   (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio);
   or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local
   backend — highest effort, Kokoro-styled output.

## Best practices

- **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern
  cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro
  fine-tune on quality and effort.
- **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract
  clean voice with Demucs if the source is music/mixed.
- **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint
  for many TTS models — it is the only server with the standard
  `/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup.
- **Check the license before serving publicly:** several strong cloners are
  non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom
  licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT /
  Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
- **Keep TTS off the GPUs.** The 2026-09-22 advice to "co-locate on the 3090 at
  `--gpu-memory-utilization ~0.35`" is retired: that card belongs to
  frank-zealot (2026-10-17) and the R9700s to frank-archon, and Gitea issue #2
  measures both as full. Run TTS on CPU; revisit only if the operator frees a
  card on purpose.
- **Guard the endpoint like any other service:** if vhosted, it goes behind the
  Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the
  internal address, never the public hostname.

## Citations

- OMP speech tool: https://omp.sh/docs/tools ("Generate speech", gated by
  `speechgen.enabled`; the old `https://omp.sh/docs/tools/tts` path returns 404
  as of 2026-10-18) · installed binary
  `@oh-my-pi/pi-coding-agent/dist/cli.js` (the `openai-speech` API, the
  15 000-character cap, the `voice_id` forwarding rule, local Kokoro WAV-only)
- Kokoro: https://github.com/hexgrad/Kokoro ·
  https://huggingface.co/spaces/hexgrad/Kokoro-TTS
- Kokoro fine-tune: https://github.com/semidark/kikiri-tts
- Chatterbox: https://github.com/resemble-ai/chatterbox
- Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS ·
  https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base
- CosyVoice: https://github.com/FunAudioLLM/CosyVoice
- GLM-TTS: https://github.com/zai-org/GLM-TTS ·
  https://huggingface.co/zai-org/GLM-TTS
- VoxCPM2: https://github.com/OpenBMB/VoxCPM ·
  https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
  https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- mainline `llama-tts`:
  https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
  Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)
- VoxCPM2 GGUFs (not published by OpenBMB):
  https://huggingface.co/DennisHuang648/VoxCPM2-GGUF
- GLM-TTS license split: model card `mit` (HF API 2026-10-18) vs code
  `LICENSE` (Apache-2.0):
  https://raw.githubusercontent.com/zai-org/GLM-TTS/main/LICENSE
- F5-TTS: https://github.com/SWivid/F5-TTS
- GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS
- Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base
- IndexTTS: https://github.com/index-tts/index-tts
- Fish Speech: https://github.com/fishaudio/fish-speech
- MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano
- XTTS: https://github.com/coqui-ai/TTS
- MegaTTS3: https://github.com/bytedance/MegaTTS3
- Higgs Audio: https://github.com/boson-ai/higgs-audio
- Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2
- Spark-TTS: https://github.com/SparkAudio/Spark-TTS
- Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603
- Parler-TTS: https://github.com/huggingface/parler-tts
- VibeVoice: https://github.com/microsoft/VibeVoice
- vLLM-Omni: https://github.com/vllm-project/vllm-omni ·
  https://docs.vllm.ai/projects/vllm-omni/en/latest/
- Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI
- whisper.cpp: https://github.com/ggml-org/whisper.cpp ·
  faster-whisper: https://github.com/SYSTRAN/faster-whisper ·
  whisperX: https://github.com/m-bain/whisperX
- Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b ·
  Canary: https://huggingface.co/nvidia/canary-1b-v2
- Moonshine: https://github.com/moonshine-ai/moonshine
- Silero VAD: https://github.com/snakers4/silero-vad
- xAI custom voices:
  https://docs.x.ai/developers/model-capabilities/audio/custom-voices