dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
chore/pr-attribution-trailer
serverAppConfigs/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
T
Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages) 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00

18 KiB
Raw Permalink Blame History

type, title, description, tags, timestamp
type title description tags timestamp
research Local TTS and Voice Cloning How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
research
tts
voice-cloning
audio
stt
vllm-omni
kokoro
2026-10-18T00:00:00Z

Local TTS and Voice Cloning

Research brief (2026-09-22) on creating a new local TTS voice from sample files, the open-weight voice-cloning landscape, the full local AI audio stack, and a recommended deployment for the fleet. Every claim is cited in Citations. The fleet has no audio input today (that capability left with the retired NPU endpoint, returned with the Gemma4 audio projector on 2026-09-17, and left again with that retirement on 2026-10-01 — audio is still greenfield: every fleet endpoint declares input: [text, image] (~/.omp/agent/models.yml:63,149,214).

Summary

  • Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice. Voices are a fixed pack. A community fine-tune recipe exists (semidark/kikiri-tts): a ~51 h multi-speaker base stage + a few hundred single-speaker samples, then voice-pack extraction. Feasible on one GPU, but the output is Kokoro-styled, and OMP's local backend only accepts its built-in voice list (a custom pack needs a patch).
  • The 2026 open-weight cloning landscape is strong. Permissive-licensed leaders for a single-GPU homelab: Chatterbox (MIT), Qwen3-TTS (Apache-2.0), CosyVoice 3 (Apache-2.0), GLM-TTS (MIT weights), VoxCPM2 (Apache-2.0), F5-TTS (MIT), GPT-SoVITS (MIT).
  • OMP integration (re-verified 2026-10-18): the speech tool is gated by speechgen.enabled and resolves models through the speech role and its fallback chain (modelRoles.speech; this workstation runs local/kokoro). Three APIs: local Kokoro ONNX (always WAV/PCM16), xai-tts, and openai-speech — a generic OpenAI-compatible /v1/audio/speech client (MP3 or WAV, voice_id forwarded only when explicitly set, 15 000 characters max; DeepInfra is now just a model on that API). The 2026-09-22 finding that OMP has no configurable OpenAI TTS endpoint is retired: a local server needs a speech-role entry, not a patch.
  • Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090. Gitea issue #2 rules out every GPU option (the 3090 has carried frank-zealot since 2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF engine on 8880 behind a voice-registry shim on 8890 (scripts/voiceshim.env, dankclaw/voiceservice/) that speaks OpenAI /v1/audio/speech and resolves voices by name. Nothing is deployed yet — on 2026-10-18 voiceshim has no unit installed and nothing answers on 8880 or 8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. No RTF or quality number has been measured for this shape, so this page states none.

Can Kokoro learn a new voice?

Kokoro (hexgrad/Kokoro, Apache-2.0, 82 M params, non-autoregressive) ships fixed voice packs — there is no reference-audio path, so zero-shot cloning is not an option. OMP's local backend runs onnx-community/Kokoro-82M-v1.0-ONNX offline, choosing a voice from a built-in list (tts.localVoice).

The best-documented fine-tune path is semidark/kikiri-tts (Apache-2.0):

Stage Data Purpose
Multi-speaker base ~51 h audio, 51 speakers language adaptation / base fine-tune
Single-speaker 455–627 samples (~1–2 h of one voice) voice specialization
Extraction — voice-pack (.pt) extraction for inference

Verdict: fine-tuning Kokoro to a personal voice is possible on a 3090, but you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe, and patch OMP to load the custom pack. The effort/quality trade is worse than running a modern zero-shot cloner on the same GPU.

2026 voice-cloning landscape (permissive-licensed, single-GPU)

Ranked (2026-09-22) for a 3090 homelab, English-primary, personal voice; the 2026-10-18 fleet decision below moved this work off the GPU entirely:

  1. Chatterbox (MIT) — best license/quality/compute ratio; ~10 s reference (the README's own example is your_10s_ref_clip.wav); 110 M (Nano) to 0.5 B (Turbo/Multilingual); ~1–2 GB VRAM (unverified 2026-10-18); Nano is documented for CPU — "3x faster than realtime on 8 CPU cores". Top pick for a personal English voice. No server in the README (library use only; wrap it yourself); every generated file carries Resemble's PerTh watermark.
  2. Qwen3-TTS-1.7B-Base (Apache-2.0) — 3 s cloning, 10 languages, streaming, vLLM-Omni serving with a POST /v1/audio/voices upload (unverified 2026-10-18 — the reachable docs do not list it), the closest thing to "named voice" semantics, official fine-tuning code. Top pick for a OpenAI-compatible deployment.
  3. CosyVoice 3 (Apache-2.0) — 0.5 B, 9 languages + 18 zh dialects, 150 ms bi-streaming, FastAPI and gRPC servers.
  4. GLM-TTS (MIT model weights, Apache-2.0 code) — CER 0.89 after multi-reward GRPO (down from 1.03), RL emotion control, zh/en; the prompt audio shipped in the repo is marked research-only.
  5. VoxCPM2 (Apache-2.0) — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min of audio, ~8 GB VRAM. Best "fine-tune my voice" option.
  6. F5-TTS (MIT) — 0.3–0.5 B, en/zh; Docker + Triton.
  7. GPT-SoVITS (MIT) — best few-shot fine-tune UX (1 min of data), WebUI, Docker images.

Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS (non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE encoder not released); Voxtral TTS (no official cloning — codec encoder not released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS (code withdrawn 2025-09).

OMP integration (definitive)

The tool ships hidden and is enabled by speechgen.enabled (~/.omp/agent/config.yml:222-223); the harness docs list it as "Generate speech". Models come from the speech role and its fallback chain, not from a hardcoded backend list (modelRoles.speech, ~/.omp/agent/config.yml:16 = local/kokoro today):

API Endpoint Notes (from the installed bundle, 2026-10-18)
local-inference on-device ONNX, onnx-community/Kokoro-82M-v1.0-ONNX (Apache-2.0) no network, always WAV/PCM16, voice from the built-in list (tts.localVoice)
xai-tts POST https://api.x.ai/v1/tts ara/eve/leo/rex/sal + custom voice IDs; MP3 or WAV
openai-speech POST <provider baseUrl>/audio/speech the generic OpenAI-compatible client — MP3 or WAV, voice_id forwarded only when explicitly set, 15 000-character cap on input; DeepInfra is one model entry on it (https://api.deepinfra.com/v1/openai)

The 2026-09-22 finding — "no generic OpenAI-compatible TTS endpoint … deepinfra is a hardcoded URL, not a configurable base URL" — is retired: openai-speech is that configurable client, selected by putting a model on the speech role. Still true from the original audit: the local Kokoro backend accepts only its built-in voice list, so a custom pack needs a patch.

Paths for a Frank-cloned voice (2026-10-18):

  1. Local server + speech role, no patch. Register the local engine (or the 8890 voice shim) as a provider model on the openai-speech API and point modelRoles.speech at it; the shim turns a voice name into a reference clip and forwards it as base64 reference_audio (scripts/voiceshim.env:19-35). Not deployed as of 2026-10-18.
  2. xAI console custom voice — works today, zero code. Record 2–3 min clean speech (≤120 s, 24 kHz 16-bit mono WAV) → console → voice_id → speech tool. Free (≤30 voices), cloud, US-only.
  3. No-patch workaround. The bash tool can curl the local TTS server ad hoc, and the harness tts device writes a speech file to a path you give it.
  4. Kokoro fine-tune → OMP local backend. Highest effort, Kokoro-styled.

Full local AI audio stack

STT / ASR: whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word timestamps + diarization); NVIDIA Parakeet TDT 1.1B (CC-BY-4.0, LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 / Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT, 60-min, 50+ langs).

VAD: Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in faster-whisper/whisperX).

TTS serving: vLLM-Omni (vllm-project/vllm-omni, Apache-2.0) publishes the official images vllm/vllm-omni (CUDA) and vllm/vllm-omni-rocm (ROCm), both tagged v0.28.0 (Docker Hub tag lists, 2026-10-18), and serves Qwen3-TTS (docs/models/supported_models.md). It serves the OpenAI speech path POST /v1/audio/speech; the POST /v1/audio/voices upload route is (unverified 2026-10-18) — the docs reachable today do not document it. Its gfx942/gfx950 validation note is a footnote scoped to MiniMax H3 diffusion, not to TTS, and the fleet runs no vLLM anywhere: the R9700s carry radiance as frank-archon (scripts/serve-frank-archon.sh:2). It is also no longer "the one server" with the OpenAI shape — llama.cpp's own llama-tts-server has it too (next section). Others: Kokoro-FastAPI (Docker, :8880, Apache-2.0), CosyVoice FastAPI and gRPC (runtime/python/fastapi, runtime/python/grpc; the README documents no WebSocket server), F5-TTS Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 tools/api_server.py (own format, not OpenAI-shaped).

Post-processing: SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the reference-clip standard); Demucs (htdemucs) for vocal separation before cloning; the Perth watermark is built into Chatterbox.

llama.cpp-native TTS serving (verified 2026-10-18)

The CPU shape does not need a Python server at all:

  • llama-tts-server (tc-mb/llama.cpp-omni, MIT, "Omni inference in C/C++") is a standalone OpenAI-compatible TTS server that runs without an LLM model (build target llama-tts-server). POST /v1/audio/speech takes model ("voxcpm", also accepts "voxcpm2"), input, voice — documented "reserved", so do not build named-voice semantics on it — response_format (wav/pcm), reference_audio (base64 WAV for cloning), seed 42, cfg_value 2.0, inference_timesteps 10, max_steps 200, temperature 1.0. POST /v1/audio/speech/stream returns chunked WAV; POST /v1/voxcpm2/init hot-loads a runtime from base_lm + acoustic GGUFs with n_gpu_layers (-1 = all); GET /v1/audio/speech/models lists what is loaded (tools/omni/voxcpm2/README.md:232, 321-354).
  • Mainline llama-tts (ggml-org/llama.cpp) has --tts-speaker-file for a speaker reference with Qwen3-TTS and with Kyutai Pocket-TTS — the latter requires it ("the model produces almost no audio without it"), and Pocket-TTS converts one languages/<name> directory, never the repo root (tools/tts/README.md:20-58; kyutai/pocket-tts is CC-BY-4.0).
  • VoxCPM2 GGUF provenance: openbmb/VoxCPM2 publishes PyTorch weights only (model.safetensors, no .gguf sibling — HF API 2026-10-18). The GGUFs the fork loads come from DennisHuang648/VoxCPM2-GGUF (VoxCPM2-BaseLM-F16.gguf, VoxCPM2-BaseLM-Q8_0.gguf, VoxCPM2-Acoustic-F16.gguf, Apache-2.0 card) or from the fork's own tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py.

Nothing has been measured on this fleet for any of these engines — no RTF, no latency, no MOS. That absence is the finding, not an omission.

Recommended deployment (Dankclaw)

As decided in Gitea issue #2, this stays CPU-only on Dankclaw: the RTX 3090 has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's, so no card may be asked to co-host TTS. The shape is one engine plus a thin voice shim, and nothing here is deployed as of 2026-10-18 — systemctl --user is-enabled voiceshim reports not-found and nothing answers on 8880 or 8890.

Service Image / install Port CPU Notes
TTS engine llama-tts-server (tc-mb/llama.cpp-omni) on a VoxCPM2 GGUF; vllm/vllm-omni + Qwen3-TTS only if a GPU is ever deliberately freed 8880 engine decision still open OpenAI /v1/audio/speech; n_gpu_layers is there for a GPU boot
Voice shim dankclaw/voiceservice/ (stdlib container) + scripts/voiceshim.env + dankclaw/voiceservice/quadlet/voiceshim.container 8890 yes voice name → reference.wav, voice/voice_id on the wire, mp3 re-encode; backends omni-http (forwards base64 reference_audio) and llama-tts-cli (--tts-speaker-file, nice 19)
STT (English) Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server 8882 yes CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card)
STT (multilingual, opt.) faster-whisper large-v3 int8 8883 yes MIT
VAD Silero (embedded) — yes MIT; model file 2 327 524 B

8880/8890 are what scripts/voiceshim.env:35,49 carry (TTS_UPSTREAM= http://host.containers.internal:8880, SHIM_PORT=8890); 8882-8883 for STT keep clear of the fleet's LLM ports 7770/7771.

Voice-creation workflow (the "from sample files" answer)

  1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
  2. Zero-shot: keep the clip as ref_audio — Chatterbox/Qwen3-TTS/CosyVoice/ GLM-TTS clone from it per request (no training).
  3. Dedicated voice: upload once to Qwen3-TTS via /v1/audio/voices (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio); or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
  4. Kokoro fine-tune only if the voice must live inside OMP's offline local backend — highest effort, Kokoro-styled output.

Best practices

  • Prefer zero-shot cloning over fine-tuning for a personal voice: a modern cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro fine-tune on quality and effort.
  • Normalize reference clips to 24 kHz 16-bit mono WAV before use; extract clean voice with Demucs if the source is music/mixed.
  • Serve through vLLM-Omni when you want a single OpenAI-compatible endpoint for many TTS models — it is the only server with the standard /v1/audio/speech + /v1/audio/voices shape across the 2026 lineup.
  • Check the license before serving publicly: several strong cloners are non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT / Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
  • Keep TTS off the GPUs. The 2026-09-22 advice to "co-locate on the 3090 at --gpu-memory-utilization ~0.35" is retired: that card belongs to frank-zealot (2026-10-17) and the R9700s to frank-archon, and Gitea issue #2 measures both as full. Run TTS on CPU; revisit only if the operator frees a card on purpose.
  • Guard the endpoint like any other service: if vhosted, it goes behind the Anubis → Authentik chain; non-browser clients (OMP's curl) must target the internal address, never the public hostname.

Citations

Reference in new issue View Git Blame Copy Permalink