--- type: research title: Local TTS and Voice Cloning description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment. tags: - research - tts - voice-cloning - audio - stt - vllm-omni - kokoro timestamp: 2026-10-18T00:00:00Z --- # Local TTS and Voice Cloning Research brief (2026-09-22) on creating a new local TTS voice from sample files, the open-weight voice-cloning landscape, the full local AI audio stack, and a recommended deployment for the fleet. Every claim is cited in [Citations](#citations). The fleet has no audio input today (that capability left with the retired NPU endpoint, returned with the Gemma4 audio projector on 2026-09-17, and left again with that retirement on 2026-10-01 — audio is still greenfield: every fleet endpoint declares `input: [text, image]` (`~/.omp/agent/models.yml:63,149,214`). ## Summary - **Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice.** Voices are a fixed pack. A community fine-tune recipe exists (`semidark/kikiri-tts`): a ~51 h multi-speaker base stage + a few hundred single-speaker samples, then voice-pack extraction. Feasible on one GPU, but the output is Kokoro-styled, and OMP's local backend only accepts its built-in voice list (a custom pack needs a patch). - **The 2026 open-weight cloning landscape is strong.** Permissive-licensed leaders for a single-GPU homelab: **Chatterbox (MIT)**, **Qwen3-TTS (Apache-2.0)**, **CosyVoice 3 (Apache-2.0)**, **GLM-TTS (MIT weights)**, **VoxCPM2 (Apache-2.0)**, **F5-TTS (MIT)**, **GPT-SoVITS (MIT)**. - **OMP integration (re-verified 2026-10-18):** the speech tool is gated by `speechgen.enabled` and resolves models through the **`speech` role and its fallback chain** (`modelRoles.speech`; this workstation runs `local/kokoro`). Three APIs: local Kokoro ONNX (always WAV/PCM16), `xai-tts`, and **`openai-speech` — a generic OpenAI-compatible `/v1/audio/speech` client** (MP3 or WAV, `voice_id` forwarded only when explicitly set, 15 000 characters max; DeepInfra is now just a model on that API). The 2026-09-22 finding that OMP has *no* configurable OpenAI TTS endpoint is **retired**: a local server needs a speech-role entry, not a patch. - **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue #2 rules out every GPU option (the 3090 has carried frank-zealot since 2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF engine on **8880** behind a voice-registry shim on **8890** (`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI `/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on 2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or 8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality number has been measured for this shape**, so this page states none. ## Can Kokoro learn a new voice? Kokoro ([hexgrad/Kokoro](https://github.com/hexgrad/Kokoro), Apache-2.0, 82 M params, non-autoregressive) ships **fixed voice packs** — there is no reference-audio path, so zero-shot cloning is not an option. OMP's local backend runs `onnx-community/Kokoro-82M-v1.0-ONNX` offline, choosing a voice from a built-in list (`tts.localVoice`). The best-documented fine-tune path is [`semidark/kikiri-tts`](https://github.com/semidark/kikiri-tts) (Apache-2.0): | Stage | Data | Purpose | |---|---|---| | Multi-speaker base | ~51 h audio, 51 speakers | language adaptation / base fine-tune | | Single-speaker | 455–627 samples (~1–2 h of one voice) | voice specialization | | Extraction | — | voice-pack (.pt) extraction for inference | **Verdict:** fine-tuning Kokoro to a personal voice is possible on a 3090, but you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe, and patch OMP to load the custom pack. The effort/quality trade is worse than running a modern zero-shot cloner on the same GPU. ## 2026 voice-cloning landscape (permissive-licensed, single-GPU) Ranked (2026-09-22) for a 3090 homelab, English-primary, personal voice; the 2026-10-18 fleet decision below moved this work off the GPU entirely: 1. **Chatterbox (MIT)** — best license/quality/compute ratio; ~10 s reference (the README's own example is `your_10s_ref_clip.wav`); 110 M (Nano) to 0.5 B (Turbo/Multilingual); ~1–2 GB VRAM *(unverified 2026-10-18)*; Nano is documented for CPU — "3x faster than realtime on 8 CPU cores". *Top pick for a personal English voice.* No server in the README (library use only; wrap it yourself); every generated file carries Resemble's PerTh watermark. 2. **Qwen3-TTS-1.7B-Base (Apache-2.0)** — 3 s cloning, 10 languages, streaming, vLLM-Omni serving with a `POST /v1/audio/voices` upload *(unverified 2026-10-18 — the reachable docs do not list it)*, the closest thing to "named voice" semantics, official fine-tuning code. *Top pick for a OpenAI-compatible deployment.* 3. **CosyVoice 3 (Apache-2.0)** — 0.5 B, 9 languages + 18 zh dialects, 150 ms bi-streaming, FastAPI and gRPC servers. 4. **GLM-TTS (MIT model weights, Apache-2.0 code)** — CER 0.89 after multi-reward GRPO (down from 1.03), RL emotion control, zh/en; the prompt audio shipped in the repo is marked research-only. 5. **VoxCPM2 (Apache-2.0)** — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min of audio, ~8 GB VRAM. *Best "fine-tune my voice" option.* 6. **F5-TTS (MIT)** — 0.3–0.5 B, en/zh; Docker + Triton. 7. **GPT-SoVITS (MIT)** — best few-shot fine-tune UX (1 min of data), WebUI, Docker images. Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS (non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE encoder not released); Voxtral TTS (no official cloning — codec encoder not released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS (code withdrawn 2025-09). ## OMP integration (definitive) The tool ships hidden and is enabled by `speechgen.enabled` (`~/.omp/agent/config.yml:222-223`); the harness docs list it as "Generate speech". Models come from the **`speech` role and its fallback chain**, not from a hardcoded backend list (`modelRoles.speech`, `~/.omp/agent/config.yml:16` = `local/kokoro` today): | API | Endpoint | Notes (from the installed bundle, 2026-10-18) | |---|---|---| | local-inference | on-device ONNX, `onnx-community/Kokoro-82M-v1.0-ONNX` (Apache-2.0) | no network, **always WAV/PCM16**, voice from the built-in list (`tts.localVoice`) | | `xai-tts` | `POST https://api.x.ai/v1/tts` | `ara`/`eve`/`leo`/`rex`/`sal` + custom voice IDs; MP3 or WAV | | `openai-speech` | `POST /audio/speech` | the generic OpenAI-compatible client — MP3 or WAV, `voice_id` forwarded **only when explicitly set**, 15 000-character cap on `input`; DeepInfra is one model entry on it (`https://api.deepinfra.com/v1/openai`) | The 2026-09-22 finding — "**no generic OpenAI-compatible TTS endpoint** … `deepinfra` is a hardcoded URL, not a configurable base URL" — is **retired**: `openai-speech` is that configurable client, selected by putting a model on the `speech` role. Still true from the original audit: the local Kokoro backend accepts only its built-in voice list, so a custom pack needs a patch. Paths for a Frank-cloned voice (2026-10-18): 1. **Local server + speech role, no patch.** Register the local engine (or the 8890 voice shim) as a provider model on the `openai-speech` API and point `modelRoles.speech` at it; the shim turns a voice *name* into a reference clip and forwards it as base64 `reference_audio` (`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18. 2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech tool. Free (≤30 voices), cloud, US-only. 3. **No-patch workaround.** The `bash` tool can `curl` the local TTS server ad hoc, and the harness `tts` device writes a speech file to a path you give it. 4. **Kokoro fine-tune → OMP local backend.** Highest effort, Kokoro-styled. ## Full local AI audio stack **STT / ASR:** whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word timestamps + diarization); **NVIDIA Parakeet TDT 1.1B** (CC-BY-4.0, LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 / Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT, 60-min, 50+ langs). **VAD:** Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in faster-whisper/whisperX). **TTS serving:** **vLLM-Omni** ([vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni), Apache-2.0) publishes the official images `vllm/vllm-omni` (CUDA) and `vllm/vllm-omni-rocm` (ROCm), both tagged `v0.28.0` (Docker Hub tag lists, 2026-10-18), and serves Qwen3-TTS (`docs/models/supported_models.md`). It serves the OpenAI speech path `POST /v1/audio/speech`; the `POST /v1/audio/voices` upload route is *(unverified 2026-10-18)* — the docs reachable today do not document it. Its gfx942/gfx950 validation note is a footnote scoped to MiniMax H3 diffusion, **not** to TTS, and the fleet runs no vLLM anywhere: the R9700s carry radiance as frank-archon (`scripts/serve-frank-archon.sh:2`). It is also no longer "the one server" with the OpenAI shape — llama.cpp's own `llama-tts-server` has it too (next section). Others: Kokoro-FastAPI (Docker, :8880, Apache-2.0), CosyVoice **FastAPI and gRPC** (`runtime/python/fastapi`, `runtime/python/grpc`; the README documents no WebSocket server), F5-TTS Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 `tools/api_server.py` (own format, not OpenAI-shaped). **Post-processing:** SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the reference-clip standard); Demucs (`htdemucs`) for vocal separation before cloning; the Perth watermark is built into Chatterbox. ## llama.cpp-native TTS serving (verified 2026-10-18) The CPU shape does not need a Python server at all: - **`llama-tts-server`** ([`tc-mb/llama.cpp-omni`](https://github.com/tc-mb/llama.cpp-omni), MIT, "Omni inference in C/C++") is a standalone OpenAI-compatible TTS server that runs without an LLM model (build target `llama-tts-server`). `POST /v1/audio/speech` takes `model` (`"voxcpm"`, also accepts `"voxcpm2"`), `input`, `voice` — documented **"reserved"**, so do not build named-voice semantics on it — `response_format` (`wav`/`pcm`), `reference_audio` (base64 WAV for cloning), `seed` 42, `cfg_value` 2.0, `inference_timesteps` 10, `max_steps` 200, `temperature` 1.0. `POST /v1/audio/speech/stream` returns chunked WAV; `POST /v1/voxcpm2/init` hot-loads a runtime from `base_lm` + `acoustic` GGUFs with `n_gpu_layers` (`-1` = all); `GET /v1/audio/speech/models` lists what is loaded (`tools/omni/voxcpm2/README.md`:232, 321-354). - **Mainline `llama-tts`** (ggml-org/llama.cpp) has `--tts-speaker-file` for a speaker reference with **Qwen3-TTS** and with **Kyutai Pocket-TTS** — the latter requires it ("the model produces almost no audio without it"), and Pocket-TTS converts one `languages/` directory, never the repo root (`tools/tts/README.md`:20-58; `kyutai/pocket-tts` is CC-BY-4.0). - **VoxCPM2 GGUF provenance:** `openbmb/VoxCPM2` publishes **PyTorch weights only** (`model.safetensors`, no `.gguf` sibling — HF API 2026-10-18). The GGUFs the fork loads come from [`DennisHuang648/VoxCPM2-GGUF`](https://huggingface.co/DennisHuang648/VoxCPM2-GGUF) (`VoxCPM2-BaseLM-F16.gguf`, `VoxCPM2-BaseLM-Q8_0.gguf`, `VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own `tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`. Nothing has been measured on this fleet for any of these engines — no RTF, no latency, no MOS. That absence is the finding, not an omission. ## Recommended deployment (Dankclaw) As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090 has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's, so no card may be asked to co-host TTS. The shape is one engine plus a thin voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl --user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or 8890. | Service | Image / install | Port | CPU | Notes | |---|---|---|---|---| | TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot | | Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) | | STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) | | STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT | | VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B | 8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM= http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT keep clear of the fleet's LLM ports 7770/7771. ## Voice-creation workflow (the "from sample files" answer) 1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize). 2. **Zero-shot:** keep the clip as `ref_audio` — Chatterbox/Qwen3-TTS/CosyVoice/ GLM-TTS clone from it per request (no training). 3. **Dedicated voice:** upload once to Qwen3-TTS via `/v1/audio/voices` (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio); or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow. 4. **Kokoro fine-tune** only if the voice must live inside OMP's offline local backend — highest effort, Kokoro-styled output. ## Best practices - **Prefer zero-shot cloning over fine-tuning** for a personal voice: a modern cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro fine-tune on quality and effort. - **Normalize reference clips to 24 kHz 16-bit mono WAV** before use; extract clean voice with Demucs if the source is music/mixed. - **Serve through vLLM-Omni** when you want a single OpenAI-compatible endpoint for many TTS models — it is the only server with the standard `/v1/audio/speech` + `/v1/audio/voices` shape across the 2026 lineup. - **Check the license before serving publicly:** several strong cloners are non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT / Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set. - **Keep TTS off the GPUs.** The 2026-09-22 advice to "co-locate on the 3090 at `--gpu-memory-utilization ~0.35`" is retired: that card belongs to frank-zealot (2026-10-17) and the R9700s to frank-archon, and Gitea issue #2 measures both as full. Run TTS on CPU; revisit only if the operator frees a card on purpose. - **Guard the endpoint like any other service:** if vhosted, it goes behind the Anubis → Authentik chain; non-browser clients (OMP's `curl`) must target the internal address, never the public hostname. ## Citations - OMP speech tool: https://omp.sh/docs/tools ("Generate speech", gated by `speechgen.enabled`; the old `https://omp.sh/docs/tools/tts` path returns 404 as of 2026-10-18) · installed binary `@oh-my-pi/pi-coding-agent/dist/cli.js` (the `openai-speech` API, the 15 000-character cap, the `voice_id` forwarding rule, local Kokoro WAV-only) - Kokoro: https://github.com/hexgrad/Kokoro · https://huggingface.co/spaces/hexgrad/Kokoro-TTS - Kokoro fine-tune: https://github.com/semidark/kikiri-tts - Chatterbox: https://github.com/resemble-ai/chatterbox - Qwen3-TTS: https://github.com/QwenLM/Qwen3-TTS · https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base - CosyVoice: https://github.com/FunAudioLLM/CosyVoice - GLM-TTS: https://github.com/zai-org/GLM-TTS · https://huggingface.co/zai-org/GLM-TTS - VoxCPM2: https://github.com/OpenBMB/VoxCPM · https://huggingface.co/openbmb/VoxCPM2 - llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni · https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md - mainline `llama-tts`: https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md · Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0) - VoxCPM2 GGUFs (not published by OpenBMB): https://huggingface.co/DennisHuang648/VoxCPM2-GGUF - GLM-TTS license split: model card `mit` (HF API 2026-10-18) vs code `LICENSE` (Apache-2.0): https://raw.githubusercontent.com/zai-org/GLM-TTS/main/LICENSE - F5-TTS: https://github.com/SWivid/F5-TTS - GPT-SoVITS: https://github.com/RVC-Boss/GPT-SoVITS - Dots.TTS: https://huggingface.co/dots-studio/dots.tts-base - IndexTTS: https://github.com/index-tts/index-tts - Fish Speech: https://github.com/fishaudio/fish-speech - MOSS-TTS-Nano: https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Nano - XTTS: https://github.com/coqui-ai/TTS - MegaTTS3: https://github.com/bytedance/MegaTTS3 - Higgs Audio: https://github.com/boson-ai/higgs-audio - Breeze-TTS-2: https://huggingface.co/BreezeBlue/Breeze-TTS-2 - Spark-TTS: https://github.com/SparkAudio/Spark-TTS - Voxtral TTS: https://huggingface.co/MistralAI/Voxtral-4B-TTS-2603 - Parler-TTS: https://github.com/huggingface/parler-tts - VibeVoice: https://github.com/microsoft/VibeVoice - vLLM-Omni: https://github.com/vllm-project/vllm-omni · https://docs.vllm.ai/projects/vllm-omni/en/latest/ - Kokoro-FastAPI: https://github.com/remsky/Kokoro-FastAPI - whisper.cpp: https://github.com/ggml-org/whisper.cpp · faster-whisper: https://github.com/SYSTRAN/faster-whisper · whisperX: https://github.com/m-bain/whisperX - Parakeet: https://huggingface.co/nvidia/parakeet-tdt-1.1b · Canary: https://huggingface.co/nvidia/canary-1b-v2 - Moonshine: https://github.com/moonshine-ai/moonshine - Silero VAD: https://github.com/snakers4/silero-vad - xAI custom voices: https://docs.x.ai/developers/model-capabilities/audio/custom-voices