82 lines
19 KiB
Markdown
82 lines
19 KiB
Markdown
# Open-weight zero-shot cloning TTS — landscape report (scout TtsLandscape2026, recovered from aborted yield)
|
||
|
||
> State of open-weight zero-shot voice cloning TTS (as of mid/late 2026): Breeze TTS 2 (1,215 Elo, non-commercial, 3B) tops the Artificial Analysis Speech Arena; for a homelab with a 14B GGUF on a 24GB 3090, the realistic clone candidates that fit alongside the LLM are F5-TTS, CosyVoice 3, Chatterbox, IndexTTS-2.5, Qwen3-TTS 1.7B, and VoxCPM2; the biggest server is vLLM-Omni (OpenAI /v1/audio/speech, Qwen3-TTS has a voice-upload registry). XTTS v2 still works personally via the idiap fork but its weights are permanently non-commercial. Strix Halo 395 has measured llama.cpp numbers (Vulkan > ROCm) and 128GB unified memory makes coexistence trivial, though gfx1151 needs ROCm 7.x for PyTorch-based engines.
|
||
|
||
## 1. Pinggy ranking (Aug 2026, AI Speech Arena Elo) [https://pinggy.io/blog/best_open_source_self_hosted_text_to_speech_models/]
|
||
|
||
| # | Model | Elo | License | Cloning? | Hardware note |
|
||
|---|-------|-----|---------|----------|---------------|
|
||
| 1 | Breeze TTS 2 (BreezeBlue, 3B) | 1,215 | Weights: non-commercial (code Apache 2.0) | Yes, + voice design/direction | 12GB min (~7.7GiB eager), 24GB fast path; RTF 0.32; <40ms TTFA (H100) |
|
||
| 2 | Fish Audio S2 Pro (4B+400M) | 1,125 | Non-commercial (Fish Research License; arXiv paper CC BY 4.0 is the paper's license, not weights') | Yes | SGLang; RTF 0.195, ~100ms TTFA on 1 H200; no published VRAM floor |
|
||
| 3 | Step Audio EditX (3B) | 1,102 | **Apache 2.0** (highest-ranked commercially usable) | Zero-shot | 12GB min (tested on L40S); vLLM; clips <30s; 6 languages; last commit April 2026 |
|
||
| 4 | Voxtral TTS (Mistral, ~4B) | 1,082 | CC BY-NC 4.0 | Yes, ~3s, no transcript needed | 16GB VRAM; vLLM Omni 0.18.0+; RTF 0.103 c1 / 0.302 c32 on H200 |
|
||
| 5 | Magpie-Multilingual 357M (NVIDIA) | 1,066 (v2602) | NVIDIA Open Model License (commercial OK) | No (explicitly removed) | 9 languages; 20s/inference cap; 5 fixed speakers |
|
||
| 6 | Kokoro 82M | 1,060 | Apache 2.0 | No | <2GB VRAM or CPU ~4GB RAM (ONNX); 54 preset voices (8 langs per pinggy, 9 per offlinetts guide) |
|
||
| 7 | Maya1 (3B) | 1,045 | Apache 2.0 | No (text-described voice design) | 16GB+; claimed sub-100ms streaming; English only |
|
||
| 8 | Higgs Audio V3 (4B) | 1,042 | Code Apache 2.0 / weights non-commercial (Boson) | Yes, 8k-token context | SGLang-Omni ~15 req/s, RTF 0.26 (H100); no VRAM floor; 100+ languages |
|
||
| 9 | Chatterbox (0.5B) | 1,020 | **MIT** | Yes, ~5s | 23 langs (v3 ~25); Turbo variant 350M, 75ms, ~6x RTF; no VRAM floor; PerTh neural watermark on all output |
|
||
| 10 | Zonos-v0.1 (1.6B) | 1,000 | Apache 2.0 | Yes, 10-30s sample | 6GB+; ~2x RTF on 4090; hybrid needs RTX 3000+; still beta |
|
||
|
||
Unranked but important: **Qwen3-TTS** (0.6B/1.7B, Apache 2.0, 10 langs, 3s cloning, 1.7B Base ~3M downloads, CustomVoice ~2.4M; no Elo score so it isn't lineable up against Breeze/Fish), **VoxCPM2** (2B, Apache 2.0, 30 langs + 9 dialects, 48kHz, ~8GB VRAM RTF 0.30 on 4090, 1.84% WER en, 0.97% CER zh-hard, llama.cpp-omni GGUF runs on CPU with no PyTorch), **GLM-TTS** (MIT weights, RL-trained, 0.89 CER, clone from 3-10s, repo quiet since Apr 2026). Two traps: **VibeVoice 7B** was MIT-licensed Aug 2025 but withdrawn 10 days later for 'inconsistent use' — community mirrors are lawful (license can't be revoked), 969 Elo persists; don't confuse with VibeVoice-ASR (speech recognition). **There is no DeepSeek TTS** (V4 is text-only, MIT, April 2026). Two leaderboards: Provider Voices (model+makers' voices; Breeze #1 at 1215) vs Controlled Voice (same 8 cloned voices, so it measures cloning ability; Voxtral #1 at 1010, then Breeze 1002, Fish 1000) — 'Breeze's own voices are excellent; its cloning is ordinary'.
|
||
|
||
## 2. Top cloning contenders — clone requirements, license, size, 2026 quality/latency
|
||
|
||
- **F5-TTS (SWivid, v1, 330M)** [https://github.com/SWivid/F5-TTS][https://huggingface.co/api/models/SWivid/F5-TTS]: zero-shot clone from **5s** reference (per offlinetts guide; repo supports `--ref_audio`+`--ref_text`, transcript optional via ASR, which costs extra GPU memory). **License split (the same trap pinggy warns about): repo badge says MIT, but the HF model card for the weights says `license:cc-by-nc-4.0` (non-commercial)** — the weights' license is the binding one; personal use fine, commercial not. L20 (24GB, near-3090 class) measured: RTF 0.0394 at concurrency 2, 253ms avg latency (client-server Triton-TRT-LLM); PyTorch offline RTF 0.1467. 4 languages (EN/ZH/JA/KO).
|
||
- **Chatterbox (Resemble AI, 0.5B, v3 June 2026)** [https://www.resemble.ai/learn/models/chatterbox; pinggy]: clone from **~5s**; **MIT**; 22-25 languages; ~200ms latency (Turbo 75ms, ~6x RTF); blind win rates published incl. losses: beats ElevenLabs Turbo v2.5 65.3%, VibeVoice 7B 59.1%, ties Cartesia Sonic 3 49.8%; every output carries PerTh neural watermarking. Repo last commit July 2026; 26k stars / 1.8M downloads.
|
||
- **IndexTTS-2.5 (0.8B, Aug 10 2026)** [https://github.com/index-tts/index-tts][https://raw.githubusercontent.com/index-tts/index-tts/main/LICENSE]: clone from **a single reference clip** (no explicit length documented). License = 'bilibili Model Use License': royalty-free; commercial only if you exceed 100M MAU or 1B RMB revenue in the prior period — i.e. personal/home use unrestricted, commercial also effectively fine for a homelab. 5 languages (ZH/EN/JA/ES/AR), emotion + speed + pinyin/phoneme control; 'production deployment supported via vLLM' recipe (https://recipes.vllm.ai/IndexTeam/IndexTTS-2.5) plus in-repo Triton TRT server. CV3-Eval: SS 77.1-77.9 (best speaker similarity in the table), WER 3.93-5.12.
|
||
- **CosyVoice 3 (Fun-CosyVoice3-0.5B-2512, Alibaba, Dec 2025)** [https://github.com/QwenAudio/CosyVoice][https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512]: **Apache 2.0** (repo + HF card); zero-shot clone — vLLM-Omni requires `ref_audio`+`ref_text` on every request, no voice presets. Reference clip ~3-10s [inference from DeepWiki's description of upstream zero-shot mode + r/LocalLLaMA 'up to 10s']; 9 languages + 18 Chinese dialects + cross-lingual clone; streaming as low as 150ms; 0.5B params (very small); eval table: 0.81% CER zh / 1.68% WER en (RL checkpoint), SS 75-78. H200-specific optimized deploy profile needs 140GiB device memory; generic profile for smaller GPUs [https://raw.githubusercontent.com/vllm-project/vllm-omni/main/docs/user_guide/examples/online_serving/text_to_speech.md].
|
||
- **Qwen3-TTS (0.6B/1.7B, Alibaba, Jan 22 2026)** [https://github.com/QwenLM/Qwen3-TTS]: **Apache 2.0** (weights+tokenizers); Base variants do **3-second rapid voice clone** (ref_audio+ref_text, or `x_vector_only_mode=True` skips transcript, quality reduced); also VoiceDesign (text-described voice, no reference) and CustomVoice (9 premium speakers). 10 languages, trained 5M+ hours; end-to-end streaming latency as low as 97ms; benchmark: Seed-TTS WER 0.77/1.24 (12Hz 1.7B/0.6B) — claims lowest WER among the table vs MiniMax and ElevenLabs. 1.7B Base weights 4.54GB, 0.6B 2.52GB; recommends FlashAttention-2 to cut VRAM (bf16, so ~6-8GB total on a 1.7B run is a reasonable estimate [inference]). vLLM support officially day-0 but 'only offline inference' per the repo at read time; current vLLM-Omni docs list Qwen3-TTS with PCM+WebSocket streaming and a **voice registry upload endpoint**.
|
||
- **Voxtral TTS (Mistral, ~4B, Mar 23 2026)**: CC BY-NC 4.0 (non-commercial, fine for personal use); clone from **~3s with no transcript** (infers prosody/emotion from the voice prompt); best open-weight clone on the Controlled Voice Arena (1,010); 9 languages; 16GB VRAM; RTF 0.103 H200 c1, 0.302 c32. No WER/similarity published (Mistral says they prioritized human-eval).
|
||
- **Breeze TTS 2 (3B)**: non-commercial weights (written authorization from RESONIA, INC. for commercial); cloning is one of 3 modes (clone / voice-design / voice-direction); cloning is 'ordinary' on the controlled board (1,002, three-way tie); 12GB min/24GB fast path; RTF 0.32.
|
||
- **Fish Audio S2 Pro**: non-commercial; clone via ref_audio+ref_text on vLLM-Omni; 80+ languages; RTF 0.195 H200; no VRAM floor published.
|
||
- **Step Audio EditX**: Apache 2.0; zero-shot clone; 3B; 12GB; 14 emotions, 30+ styles, 20+ paralinguistic tags; 30s per-inference cap; highest-ranked Apache-2.0 model (1,102).
|
||
|
||
## 3. OpenAI-compatible /v1/audio/speech servers today
|
||
|
||
There is no Ollama equivalent for TTS (Ollama issue #11021 open since June 2025, unimplemented) [pinggy]; what exists is a standardized API shape: 'almost everything now speaks OpenAI's POST /v1/audio/speech dialect'.
|
||
|
||
- **vLLM-Omni** [https://raw.githubusercontent.com/vllm-project/vllm-omni/main/docs/user_guide/examples/online_serving/text_to_speech.md] — the de-facto standard: `vllm serve <model> --omni` exposes `POST /v1/audio/speech` plus `/v1/audio/voices`, batch and WebSocket/PCM streaming, launchable with `--omni`. Supported: AuK/AuK-Flash, CosyVoice3 (clone only, ref_audio+ref_text every request, no presets), Fish Drama 3, Fish S2 Pro (clone, ref params), Gepard-1.0, higgs-audio v2, GLM-TTS (clone, ref params), OmniVoice (offline only), **Qwen3-TTS (clone on Base; ✓ voice presets + `/v1/audio/voices` upload — the one server with a true cloned-voice registry the `voice` param resolves to)**, VoxCPM2 (clone), Voxtral TTS (presets, clone gated upstream). Sample rate per model: 44.1k Fish, 48k VoxCPM2, 22.05k Gepard, 24k others. Note: plain vLLM does NOT do TTS text->audio; that's only in the vllm-omni subproject.
|
||
- **LocalAI** [https://github.com/mudler/LocalAI; pinggy]: broadest coverage — Coqui, Piper, Qwen3-TTS, VibeVoice, Chatterbox, fish-speech; OpenAI-compatible.
|
||
- **Kokoro-FastAPI** (remsky, 5.6k stars, Apache 2.0) [https://github.com/remsky/Kokoro-FastAPI]: Dockerized OpenAI-compatible /v1/audio/speech for Kokoro-82M; voice = preset id (54 voices, 8-9 langs); 'clone-tuning' via Inno-Kokoro voicepacks — reference-adapted voicepacks, i.e. a fine-tuning-style workaround rather than raw zero-shot cloning [inference]; CPU, NVIDIA, AMD ROCm (experimental), Apple MPS images; prebuilt multi-arch.
|
||
- **Speaches** (speaches-ai, 3.7k stars, MIT) [https://github.com/speaches-ai/speaches]: OpenAI-compatible (audio endpoints + Realtime API), but TTS = Kokoro + Piper only, so no cloning; master branch quiet since April 2026 (pinggy).
|
||
- **chatterbox-tts-api** (travisvn, 690 stars, **AGPL-3.0** — watch for your stack) [https://github.com/travisvn/chatterbox-tts-api]: FastAPI OpenAI-compatible server for Chatterbox (22 langs, voice cloning); has **Voice Library Management — upload/manage/use custom voices by name** = cloned-voice registry semantics.
|
||
- **CosyVoice3-API** (hsiang-han, community) [https://github.com/hsiang-han/CosyVoice3-API]: 'Pure OpenAI-compatible TTS API powered by CosyVoice 3, zero-shot voice cloning with 3-second reference audio'.
|
||
- **F5-TTS**: no native /v1/audio/speech; ships a Triton/TRT-LLM client-server (gRPC/HTTP) in `src/f5_tts/runtime/triton_trtllm`, plus Gradio and CLI. Community wrappers not verified.
|
||
- **IndexTTS**: production deployment via the vLLM recipe (https://recipes.vllm.ai/IndexTeam/IndexTTS-2.5, cited in repo); in-repo Triton TRT server + Gradio webui. Whether the vLLM recipe exposes /v1/audio/speech specifically is unverified [inference: it's a vLLM serving recipe, so likely the standard endpoint].
|
||
- **VibeVoice**: weights withdrawn but MIT, so community servers are lawful: vibevoice-community/VibeVoice-API ('OpenAI-compatible API for VibeVoice', community), marhensa/vibevoice-realtime-openai-api (VibeVoice-Realtime-0.5B, Open WebUI config). The 7B clone-capable weights live on community mirrors.
|
||
- **XTTS v2 / OpenVoice / 'EresBiT'**: no canonical xtts-v2-fastapi server found — only personal/community wrappers (e.g. Hammad112's XTTS v2 FastAPI platform). OpenVoice v2 (MIT, 954 Elo) is dormant since 2025 (pinggy) but is the only model with a decoupled tone-color converter; no active OpenAI server for it was found. For 'EresBiT': a dedicated 2026 OpenAI-compatible EresBiT/EresBert server was **not found** in searches [searched, not found]; the nearest live things are the community eres-bert voice-conversion tools (Chinese-ecosystem, voice-clone focused) and LocalAI's Coqui backend. Treat 'EresBiT' as [inference: no canonical 2026 server; name may be community/project-specific].
|
||
|
||
**Does the OpenAI `voice` parameter map to a cloned-voice registry?** Only two flavors genuinely do: vLLM-Omni+Qwen3-TTS (presets + /v1/audio/voices upload) and chatterbox-tts-api (upload/manage voices by name). Others: voice = preset name (Kokoro-FastAPI, Speaches, VibeVoice servers, CosyVoice3's 'default'), or cloning bypasses `voice` entirely via extra `ref_audio`+`ref_text` fields (CosyVoice3, GLM-TTS, Fish on vLLM-Omni).
|
||
|
||
## 4. Hardware reality
|
||
|
||
### (a) 24GB RTX 3090 + 14B GGUF
|
||
|
||
- 14B dense at Q4-class ≈ 14e9 × ~4.8 bits / 8 ≈ **8.4-8.7GB weights; plus ~2-4GB KV cache (context dependent) and engine overhead → ~12-16GB [inference, arithmetic]**. That leaves ~8-12GB free.
|
||
- Fits with margin (VRAM estimates for small models are mostly [inference] since vendors don't publish floors; F5-TTS RTF is measured on a 24GB L20, 3090-class):
|
||
- F5-TTS (330M): ~1-3GB [inference]; measured RTF 0.0394 / 253ms latency at c2 on L20.
|
||
- CosyVoice 3 (0.5B): ~1-3GB [inference]; generic (non-H200) vLLM deploy config exists.
|
||
- Chatterbox 0.5B / 0.35B Turbo: ~1-2GB [inference; no vendor floor].
|
||
- IndexTTS-2.5 (0.8B): ~2-4GB [inference; BF16/FP16 halves VRAM per repo docs].
|
||
- Qwen3-TTS 1.7B (bf16 + FlashAttention-2): ~6-8GB [inference; 4.54GB weights documented].
|
||
- VoxCPM2 (2B): **~8GB documented, RTF 0.30 on a 4090** [pinggy] — borderline with the LLM at Q4; llama.cpp-omni GGUF can also run it CPU-side to free GPU.
|
||
- Kokoro 82M: **<2GB documented** [pinggy] — always fits; no cloning though.
|
||
- Does NOT fit: Breeze TTS 2 (7.7GiB eager / 14.4GiB fast), Voxtral (~16GB), Step EditX (12GB floor), Fish S2 Pro (no floor but 4B+400M, SGLang — expect 12-16GB+ [inference]), Higgs V3 (4B, no floor, ~12-16GB [inference]), Zonos2 (16GB+ at FP16; GGUF for CPU). Practical conclusion: on the 3090 the good cloning pairings with a 14B are F5-TTS, CosyVoice 3, Chatterbox, IndexTTS-2.5, and Qwen3-TTS 1.7B; VoxCPM2 is borderline; the top-3 Elo cloners (Breeze/Fish/Voxtral) are too big to coexist. If the LLM workload pauses, a 16GB TTS (Voxtral/Step) is also viable.
|
||
|
||
### (b) AMD Strix Halo / Ryzen AI Max+ 395 (128GB unified, gfx1151)
|
||
|
||
- No VRAM partitioning: 128GB unified is shared by CPU and 'GPU', so 14B GGUF + any TTS engine is a RAM-budget question, not a VRAM one. 14B Q4 (~8.5GB) is a small fraction of 128GB.
|
||
- Measured LLM baseline on this exact class of box (Beelink GTR9 Pro, 128GB) [https://github.com/botAGI/strix-halo-llm-benchmarks]: Qwen3.6-35B-A3B Q4_K_M, llama.cpp server — **Vulkan 15.9 ms/token (~63 tok/s) c1 vs ROCm 18.7 ms/token (~53 tok/s) c1**; TTFA 210ms (c1) → 338ms (c4) → 865ms (c8) on Vulkan; 3-hour sustained c4: 100% completion, 1.6% drift. **Vulkan beats ROCm in this config** — supports the 'ROCm spotty' premise; the ROCm backend works but isn't the fast path here.
|
||
- TTS-specific: **F5-TTS README documents exactly this gotcha**: 'RDNA 3.5 and RDNA 4 GPUs (gfx1151/gfx1201) require ROCm 7.x; ROCm 6.x causes `HIP error: invalid device function`' (issue #1236) — so PyTorch/CUDA-style engines on this APU need the ROCm 7.x torch build. Kokoro-FastAPI ships an AMD ROCm (experimental, x86_64 only) image and a CPU image; a 3090-class card note: F5-TTS's ROCm 7.x pytorch wheels (torch==2.9.1+rocm7.2) are the documented path. CosyVoice/Chatterbox/XTTS ROCm-on-gfx1151 numbers: not found in sources [searched, not found] — treat as [inference: use ROCm 7.x torch + PyTorch, or fall back to CPU/Vulkan where supported (llama.cpp-omni supports Vulkan)].
|
||
- CPU-only paths (all fine on a 395 — 128GB, 32 cores of memory bandwidth): **Kokoro: ~2.4x real-time on CPU, <4GB RAM (ONNX) [pinggy, third-party benchmark]**; **VoxCPM2: runs as GGUF on CPU/Metal/CUDA/Vulkan via llama.cpp-omni with no PyTorch at all** — 'Kokoro has been the only real CPU option in this space for a year; this is the first serious model to join it' (pinggy); F5-TTS/Chatterbox/CosyVoice CPU speed: no published numbers [inference: expect slower than GPU; F5-TTS's PyTorch RTF 0.1467 is GPU-L20, CPU would be worse].
|
||
|
||
## 5. XTTS v2 in 2026
|
||
|
||
- **Status:** Coqui the company **shut down January 2024** without relicensing anything [pinggy, cites the shutdown post]. The original coqui-ai/TTS repo is dead (~45k stars, unmaintained); the **idiap/coqui-ai-TTS fork is actively maintained** — 2.3k stars, 5,314 commits on dev, released as PyPI `coqui-tts`, v0.27.0 adds a caching mechanism for cloned voices, v0.20.0 shipped XTTSv2 with 17 languages and <200ms streaming [https://github.com/idiap/coqui-ai-TTS]. Code is MPL-2.0.
|
||
- **License (the part that matters):** code is MPL-2.0, but the **weights are under the Coqui Public Model License, 'only non-commercial use of a machine learning model and its outputs'** [coqui-ai/TTS discussions 4145/4304; pinggy; coqui/XTTS-v2 HF LICENSE]. The CPML is non-negotiable and **nobody who can sell a commercial license exists anymore** — Coqui is gone and the cpml URL is a dead 404 [pinggy]. So: fine for Frank's personal homelab agent; not a viable commercial building block (unlike e.g. Chatterbox's MIT or IndexTTS's near-unrestricted Bilibili license).
|
||
- **Capability today:** 17 languages; clone from a short clip (~3s per pinggy, ~6s per HF coqui/XTTS-v2 card and localaimaster — sources disagree on the exact figure); <200ms streaming. Quality: **920 Elo**, dead-last on the 2026 leaderboard (vs Step EditX 1,102 Apache-2.0 and Chatterbox 1,020 MIT, both newer and permissively licensed). It's a graveyard-tier model (pinggy: 'the bottom of the leaderboard is a graveyard' — XTTS is listed among four models to skip, alongside dead OpenVoice, StyleTTS2 and MetaVoice). Still, it's the most battle-tested for personal use and LocalAI ships a Coqui backend. Recommendation: pick it only for its 17-language breadth or existing integrations; otherwise Chatterbox (MIT, ~5s clone, 2026 quality) or F5-TTS (5s clone, RTF 0.04, non-commercial OK for personal use) dominate it in 2026.
|
||
|
||
## Bottom line for a /v1/audio/speech server on a 3090 next to a 14B GGUF
|
||
|
||
- Personal/cloning quality tier, fits the budget: **Qwen3-TTS 1.7B via vLLM-Omni** (Apache 2.0, 3s clone, native /v1/audio/speech + /v1/audio/voices registry, 97ms streaming, ~6-8GB [inference]) — the best match for an omp harness wanting a `voice` parameter that resolves to a saved clone; or **Chatterbox v3 via travisvn/chatterbox-tts-api** (MIT model, AGPL server, voice library upload, ~1-2GB [inference]). For minimum footprint: **F5-TTS** (5s clone, measured RTF 0.04, ~1-2GB [inference], weights CC-BY-NC = personal-use only) or **CosyVoice 3** (Apache 2.0, 0.5B, 3-10s reference). The top Elo cloners (Breeze/Fish/Voxtral) are non-commercial (OK for personal) but 12-16GB+ and won't coexist with the 14B on a 24GB card.
|
||
- On the Strix Halo 395: anything above coexists trivially with 128GB unified; use llama.cpp/Vulkan paths where available (VoxCPM2 GGUF via llama.cpp-omni), ROCm 7.x for PyTorch engines (F5-TTS explicitly requires it on gfx1151; ROCm 6.x crashes), CPU-only fallback = Kokoro (~2.4x RT) or VoxCPM2 GGUF. [inference on CPU speeds for the PyTorch models, no vendor numbers.] |