dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
93d1181fee6ec2e78873406fda9a69ac63d73a72
serverAppConfigs/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
T
Frank 93d1181fee docs: OKF v0.1 compliance pass + fact-check + 3 research briefs 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
2026-09-22 02:09:41 -07:00

13 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
research Local TTS and Voice Cloning How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
research
tts
voice-cloning
audio
stt
vllm-omni
kokoro
2026-09-22T00:00:00Z

Local TTS and Voice Cloning

Research brief (2026-09-22) on creating a new local TTS voice from sample files, the open-weight voice-cloning landscape, the full local AI audio stack, and a recommended deployment for the fleet. Every claim is cited in Citations. The fleet has no audio input today (that capability left with the retired NPU endpoint), so this is greenfield.

Summary

  • Kokoro-82M (OMP's built-in local TTS) cannot zero-shot clone a voice. Voices are a fixed pack. A community fine-tune recipe exists (semidark/kikiri-tts): a ~51 h multi-speaker base stage + a few hundred single-speaker samples, then voice-pack extraction. Feasible on one GPU, but the output is Kokoro-styled, and OMP's local backend only accepts its built-in voice list (a custom pack needs a patch).
  • The 2026 open-weight cloning landscape is strong. Permissive-licensed leaders for a single-GPU homelab: Chatterbox (MIT), Qwen3-TTS (Apache-2.0), CosyVoice 3 (Apache-2.0), GLM-TTS (MIT), VoxCPM2 (Apache-2.0), F5-TTS (MIT), GPT-SoVITS (MIT).
  • OMP integration — definitive: OMP's tts tool supports exactly three backends (providers.tts = auto|local|xai|deepinfra, verified in the installed binary): local Kokoro ONNX, xAI Grok Voice cloud, DeepInfra cloud. There is no generic OpenAI-compatible TTS endpoint in OMP today. Two real paths for a cloned voice: (a) xAI console custom voice (free, cloud, US-only) — works today; (b) patch OMP (open source) with a ~50-line OpenAI-TTS backend pointing at a local vLLM-Omni server.
  • Recommended stack: vllm/vllm-omni serving Qwen3-TTS-1.7B-Base on the 3090 (OpenAI-compatible POST /v1/audio/speech + POST /v1/audio/voices voice upload), Parakeet TDT 1.1B for English STT, Silero VAD embedded, Chatterbox as the ultra-light English-only alternative. All fit in <8 GB VRAM and co-locate with the existing LLM services.

Can Kokoro learn a new voice?

Kokoro (hexgrad/Kokoro, Apache-2.0, 82 M params, non-autoregressive) ships fixed voice packs — there is no reference-audio path, so zero-shot cloning is not an option. OMP's local backend runs onnx-community/Kokoro-82M-v1.0-ONNX offline, choosing a voice from a built-in list (tts.localVoice).

The best-documented fine-tune path is semidark/kikiri-tts (Apache-2.0):

Stage Data Purpose
Multi-speaker base ~51 h audio, 51 speakers language adaptation / base fine-tune
Single-speaker 455–627 samples (~1–2 h of one voice) voice specialization
Extraction — voice-pack (.pt) extraction for inference

Verdict: fine-tuning Kokoro to a personal voice is possible on a 3090, but you must collect ~1–2 h of clean single-speaker audio, run the two-stage recipe, and patch OMP to load the custom pack. The effort/quality trade is worse than running a modern zero-shot cloner on the same GPU.

2026 voice-cloning landscape (permissive-licensed, single-GPU)

Ranked for a 3090 homelab, English-primary, personal voice:

  1. Chatterbox (MIT) — best license/quality/compute ratio; ~10 s reference; 110–500 M; ~1–2 GB VRAM; CPU-capable (Nano). Top pick for a personal English voice. No official server (library + Gradio; wrap in FastAPI).
  2. Qwen3-TTS-1.7B-Base (Apache-2.0) — 3 s cloning, 10 languages, streaming, vLLM-Omni serving with POST /v1/audio/voices upload (closest to "named voice" semantics), official fine-tuning code. Top pick for a server-grade, OpenAI-compatible deployment.
  3. CosyVoice 3 (Apache-2.0) — 0.5 B, 9 languages + 18 zh dialects, 150 ms bi-streaming, FastAPI/WebSocket server.
  4. GLM-TTS (MIT) — best published CER (0.89), RL emotion control, zh/en.
  5. VoxCPM2 (Apache-2.0) — 30 languages, 48 kHz, LoRA fine-tune from 5–10 min of audio, ~8 GB VRAM. Best "fine-tune my voice" option.
  6. F5-TTS (MIT) — 0.3–0.5 B, en/zh; Docker + Triton.
  7. GPT-SoVITS (MIT) — best few-shot fine-tune UX (1 min of data), WebUI, Docker images.

Notable non-commercial / restricted: Higgs Audio v3, Breeze-TTS-2, Spark-TTS (non-commercial); IndexTTS (bilibili custom license), Fish Speech S2 (research license), XTTS v2 (CPML, non-commercial, vendor defunct); MegaTTS3 (WaveVAE encoder not released); Voxtral TTS (no official cloning — codec encoder not released). Excluded from cloning: Parler-TTS (style-text only), VibeVoice-TTS (code withdrawn 2025-09).

OMP integration (definitive)

OMP's tts tool (hidden by default; enable with /tts) supports exactly three backends, selected by providers.tts:

Backend Endpoint (hardcoded in binary) Voices
local on-device ONNX, onnx-community/Kokoro-82M-v1.0-ONNX built-in list (tts.localVoice)
xai POST https://api.x.ai/v1/tts ara/eve/leo/rex/sal + any custom voice ID
deepinfra POST https://api.deepinfra.com/v1/openai/audio/speech voice_id when set

There is no generic OpenAI-compatible TTS endpoint — deepinfra is a hardcoded URL, not a configurable base URL, and OMP's models.yml custom providers (LLM chat only) do not wire into the tts tool.

Paths for a Frank-cloned voice:

  1. xAI console custom voice — works today, zero code. Record 2–3 min clean speech (≤120 s, 24 kHz 16-bit mono WAV) → console → voice_id → tts tool. Free (≤30 voices), cloud, US-only.
  2. Local, full stack — small OMP patch. OMP is open source; the DeepInfra backend is a plain HTTP POST to an OpenAI-compatible speech endpoint. A ~50-line openai backend (configurable base_url + voice) lets OMP call any local server (vLLM-Omni, Kokoro-FastAPI, Chatterbox wrapper).
  3. No-patch workaround. OMP's bash tool can curl the local TTS server ad hoc.
  4. Kokoro fine-tune → OMP local backend. Highest effort, Kokoro-styled.

Full local AI audio stack

STT / ASR: whisper.cpp (MIT, tiny→large-v3 2.9 GiB, all backends, built-in VAD); faster-whisper (MIT, CTranslate2 int8, ~4× faster); whisperX (BSD-2, word timestamps + diarization); NVIDIA Parakeet TDT 1.1B (CC-BY-4.0, LibriSpeech-clean WER 1.39% — best English open ASR); Parakeet 0.6B v3 / Canary 1B v2 (multilingual); Moonshine (MIT, on-device); VibeVoice-ASR-7B (MIT, 60-min, 50+ langs).

VAD: Silero VAD (MIT, ~2 MB, <1 ms per 30 ms chunk; embedded in faster-whisper/whisperX).

TTS serving: vLLM-Omni (vllm-project/vllm-omni, official Docker vllm/vllm-omni CUDA and vllm/vllm-omni-rocm ROCm, e.g. :v0.28.0) is the one server exposing the standard OpenAI speech API (POST /v1/audio/speech + POST /v1/audio/voices upload for Qwen3-TTS) across the 2026 lineup. CUDA needs compute ≥7.0 (3090 = 8.6 ✓); ROCm validated on gfx942 (the fleet already runs vLLM on the R9700 gfx1201 ✓). Others: Kokoro-FastAPI (Docker, :8880, OpenAI-shaped), CosyVoice FastAPI/WS, F5-TTS Gradio/Triton, GPT-SoVITS WebUI/api_v2, Fish S2 tools/api_server.py (own format, not OpenAI-shaped).

Post-processing: SoX/ffmpeg (resample/normalize; 24 kHz mono 16-bit is the reference-clip standard); Demucs (htdemucs) for vocal separation before cloning; the Perth watermark is built into Chatterbox.

Recommended deployment (Dankclaw)

TTS models are 0.1–2 B (<8 GB VRAM); STT (Parakeet 1.1 B) is CPU-fast. Keep it all on Dankclaw (3090 = primary GPU, CUDA = best vLLM-Omni support); the Dankserver R9700s stay dedicated to frank-medium.

Service Image / install Port GPU Notes
TTS (primary, cloning) vllm/vllm-omni:v0.28.0 · vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8881 --gpu-memory-utilization 0.35 8881 3090 (~6 GB) OpenAI /v1/audio/speech; upload Frank's voice once via /v1/audio/voices; 3 s cloning; 10 langs; Apache-2.0
TTS (light, English) Chatterbox (MIT) in a small FastAPI container 8880 3090 (~1 GB) / CPU 10 s ref clip; wrap chatterbox.generate in ~30 lines
STT (English) Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server 8882 CPU CC-BY-4.0; WER 1.39%
STT (multilingual, opt.) faster-whisper large-v3 int8 8883 CPU / iGPU MIT
VAD Silero (embedded) — CPU MIT

Ports 8880–8883 avoid the fleet's LLM ports (7770/7771).

Voice-creation workflow (the "from sample files" answer)

  1. Record 2–3 min clean speech (24 kHz 16-bit mono WAV; ffmpeg-normalize).
  2. Zero-shot: keep the clip as ref_audio — Chatterbox/Qwen3-TTS/CosyVoice/ GLM-TTS clone from it per request (no training).
  3. Dedicated voice: upload once to Qwen3-TTS via /v1/audio/voices (vLLM-Omni) and reference by ID; or LoRA-fine-tune VoxCPM2 (5–10 min audio); or the GPT-SoVITS 1-min fine-tune for the classic WebUI workflow.
  4. Kokoro fine-tune only if the voice must live inside OMP's offline local backend — highest effort, Kokoro-styled output.

Best practices

  • Prefer zero-shot cloning over fine-tuning for a personal voice: a modern cloner (Chatterbox/Qwen3-TTS) on a 10 s–3 min reference beats a Kokoro fine-tune on quality and effort.
  • Normalize reference clips to 24 kHz 16-bit mono WAV before use; extract clean voice with Demucs if the source is music/mixed.
  • Serve through vLLM-Omni when you want a single OpenAI-compatible endpoint for many TTS models — it is the only server with the standard /v1/audio/speech + /v1/audio/voices shape across the 2026 lineup.
  • Check the license before serving publicly: several strong cloners are non-commercial (Higgs v3, Breeze-TTS-2, Spark-TTS, Voxtral) or carry custom licenses (IndexTTS, Fish S2, XTTS v2). For a personal homelab voice, MIT / Apache-2.0 (Chatterbox, Qwen3-TTS, GLM-TTS, VoxCPM2) is the safe set.
  • Co-locate on the 3090 with a low --gpu-memory-utilization (~0.35) so the TTS engine shares the card with frank-small; move to the R9700s only if TTS throughput demand grows.
  • Guard the endpoint like any other service: if vhosted, it goes behind the Anubis → Authentik chain; non-browser clients (OMP's curl) must target the internal address, never the public hostname.

Citations

Reference in New Issue View Git Blame Copy Permalink