From 0e7936f94ec59b90098b8eb8c72842b206658807 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 18 Oct 2026 16:19:51 -0700 Subject: [PATCH] docs(tts): record the measured CPU engine A/B behind frank-tts Both candidate :8880 engines were measured on Dankclaw's CPU (no GPU, no container, nothing deployed). arm A qwentts.cpp 5916bf7 resident at RTF 0.55-0.57 warm / 0.67 ICL / 1.65 cold; arm B mainline llama-tts at 0.93-0.99 with a +-0.25 spread. Records the four configuration laws the trial produced (boot-warm every voice, .spk+.rvq replay is byte-identical, streaming != buffered, /v1/audio/voices rejects a lone spk_b64), the likeness measurement's resolution limit, and re-keys the stale 8890/voiceshim shape on that page to frank-tts on 7772. The engine pick stays open: the likeness metric cannot rank clones, so the audition clips decide. --- .../Research/Local TTS and Voice Cloning.md | 144 +++++++++++++++--- 1 file changed, 119 insertions(+), 25 deletions(-) diff --git a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md index 7caf0a7..f04bd7b 100644 --- a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md +++ b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md @@ -1,7 +1,7 @@ --- type: research title: Local TTS and Voice Cloning -description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment. +description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts, the full local AI audio stack, and a recommended homelab deployment. tags: - research - tts @@ -10,7 +10,7 @@ tags: - stt - vllm-omni - kokoro -timestamp: 2026-10-18T00:00:00Z +timestamp: 2026-10-18T16:05:00Z --- # Local TTS and Voice Cloning @@ -47,13 +47,15 @@ greenfield: every fleet endpoint declares `input: [text, image]` local server needs a speech-role entry, not a patch. - **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue #2 rules out every GPU option (the 3090 has carried frank-zealot since - 2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF - engine on **8880** behind a voice-registry shim on **8890** - (`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI - `/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on - 2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or - 8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality - number has been measured for this shape**, so this page states none. + 2026-10-17; both R9700s are frank-archon's). The shape is a TTS engine on + **8880** behind **frank-tts**, the fleet's speech endpoint on **7772** + (`scripts/frank-tts.env`, `dankclaw/voiceservice/`): one OpenAI + `/v1/audio/speech` façade that resolves a voice *by name* and never shows a + path. Nothing is deployed — no `frank-tts` unit and no container exist on the + host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet + TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine behind it is measured, not + guessed** (see *Measured CPU engine A/B* below); the pick waits on the + operator's ears. ## Can Kokoro learn a new voice? @@ -133,11 +135,11 @@ accepts only its built-in voice list, so a custom pack needs a patch. Paths for a Frank-cloned voice (2026-10-18): -1. **Local server + speech role, no patch.** Register the local engine (or the - 8890 voice shim) as a provider model on the `openai-speech` API and point - `modelRoles.speech` at it; the shim turns a voice *name* into a reference - clip and forwards it as base64 `reference_audio` - (`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18. +1. **Local server + speech role, no patch.** Register frank-tts (port 7772) as + a provider model on the `openai-speech` API and point `modelRoles.speech` at + it; the façade turns a voice *name* into that pack's reference clip and + forwards it as base64 `reference_audio` (`scripts/frank-tts.env`, + `TTS_BACKEND=omni-http`). Not deployed as of 2026-10-18. 2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech tool. Free (≤30 voices), cloud, US-only. @@ -206,29 +208,117 @@ The CPU shape does not need a Python server at all: `VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own `tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`. -Nothing has been measured on this fleet for any of these engines — no RTF, no -latency, no MOS. That absence is the finding, not an omission. +Arm B (`llama-tts`) and arm A (`qwentts.cpp`) have both been measured on this +box — RTF, first-audio latency and RSS are in the next section. The other +servers in the table above (vLLM-Omni, Kokoro-FastAPI, CosyVoice, F5-TTS, +GPT-SoVITS) remain unmeasured here; that absence is still the finding. + +## Measured CPU engine A/B (2026-10-18) + +Both candidate upstreams for frank-tts were measured on Dankclaw's CPU — 8 +threads, no GPU touched, three runs per arm, seed 42, identical text, against +the fleet's own audition voice packs. + +- **Arm A — [`ServeurpersoCom/qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp)** + at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (2026-10-16), weights + from [`Serveurperso/Qwen3-TTS-GGUF`](https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF): + `qwen-talker-1.7b-base-Q4_K_M.gguf` (1,219,245,248 B), + `qwen-talker-0.6b-base-Q4_K_M.gguf` (628,905,056 B), + `qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B) — byte counts verified + against the Hub API. Builds with zero compiler warnings; `GGML_BLAS=OFF` is + forced on this host (only `libopenblas.so.0` exists, no dev headers, so the + vendor `buildcpu.sh` cannot configure). +- **Arm B — mainline `llama-tts`** with `--tts-speaker-file`, i.e. the + `llama-tts-cli` backend `scripts/frank-tts.env` already implements. + +RTF is **seconds of compute per second of audio** (lower is better; below 1 is +faster than real time). + +| Arm | RTF | Run spread | Peak RSS | +|---|---|---|---| +| A · 0.6 B talker, CLI | 0.44 | ±0.002 | 3.4 GiB | +| A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB | +| A · 1.7 B, CLI, embedding-only clone | 0.63 | ±0.02 | n/m | +| A · 1.7 B, CLI, ICL clone | 0.92 | ±0.01 | n/m | +| A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB | +| A · same, ICL voice warm / **cold (first request)** | 0.67 / **1.65** | one-shot | 5.7 GiB | +| B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m | + +**Read RTF only within an arm.** Conditioning changes clip length — the same +seed yields 18.00 s of audio embedding-only and 15.12 s under ICL — so the +denominator is the arm's own audio duration. + +Arm A's resident server is ~1.7× faster than arm B and about 40× tighter in +run-to-run spread, and it already serves the HTTP shape frank-tts's façade +speaks: `/health`, `/v1/models`, `/v1/audio/voices` (GET/POST/DELETE), +`/v1/audio/speech` (`wav`/`pcm`), chunked streaming with the first audio byte +at 86–451 ms and a server-side time-to-first-audio of 94–112 ms. Arm B has no +HTTP path at all — that is what the façade's CLI backend exists for. + +Four laws came out of the trial, each worth a line of configuration: + +- **Boot-warm every voice.** The first request for a registered voice decodes + the whole reference to snapshot codec state: 6.2–8.0 s, first audio byte at + 5,530 ms. Later requests restore that state in 0.5 ms. An engine unit must + run one throwaway synthesis per voice at start, or the first request of the + day pays five seconds. +- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference + WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T, + 11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a + **byte-identical** WAV and saves 1.4 s per request. That pair is the + engine-artifact term the voice-pack law in `scripts/frank-tts.env` reserves a + slot for — arm A pins it as `.spk` + `.rvq` beside `reference.wav`. +- **Streaming is not bit-identical to buffered**, whatever the upstream README + claims: same text, seed and voice, 92.2 % of samples differ (peak delta + 1139/32768, −41.6 dB RMS — inaudible). A fixed seed does not pin output + across `response_format`, so never mix the two paths in a cache or a golden + fixture. +- **A voice cannot be registered from an embedding alone**: `POST + /v1/audio/voices` accepts `wav_b64` or **both** `spk_b64` and `rvq_b64` + (`src/tts-server.h:480`), so an embedding-only voice has to be sent as the + WAV, or the lone-`spk` path has to be upstreamed. + +**Likeness is not measurable at this resolution.** x-vector cosine against each +speaker's own reference: ICL 0.9886 / 0.9901, embedding-only 0.9844 / 0.9812, +uncloned 0.9228 — but **two different speakers score 0.9497** on the same +metric. The whole clone effect lives in a 0.04-wide band and the +ICL-over-embedding gap is inside noise, so the four audition clips +(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3` +mainline partner, guide in `LISTEN-engine.md`) are the instrument, and **the +engine pick stays open until the operator has heard them.** The cosine +extractor is still worth keeping: it is engine-neutral (it scores mainline +output through the same encoder) and it separates an uncloned voice cleanly. + +Two operational details. The engine takes 8 threads and exposes **no `-t` +flag** — `src/backend.h` uses `hardware_concurrency()/2` and prints the choice +at load, so pinning a cpuset on the quadlet changes that number implicitly. And +ICL needs a reference transcript that did not exist: the fleet has no ASR +endpoint and no audio input, so the crops were transcribed with +`vosk-small-en-us` inside the work dir, and two of three transcripts came back +suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into +the pack manifest instead of transcribing on the host. ## Recommended deployment (Dankclaw) As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090 has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's, -so no card may be asked to co-host TTS. The shape is one engine plus a thin -voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl ---user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or -8890. +so no card may be asked to co-host TTS. The shape is one engine plus one +façade, and **nothing here is deployed as of 2026-10-18**: `frank-tts` has +neither a unit nor a container on the host, and nothing listens on 7772 or +8880. | Service | Image / install | Port | CPU | Notes | |---|---|---|---|---| -| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot | -| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) | +| TTS engine | arm A `qwentts.cpp` `tts-server` on the Qwen3-TTS GGUFs, or arm B mainline `llama-tts`; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself; arm B has no HTTP path and runs through the façade's CLI backend | +| frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices//reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) | | STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) | | STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT | | VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B | -8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM= -http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT -keep clear of the fleet's LLM ports 7770/7771. +7772 is frank-tts's host port — the next slot in the fleet's model-endpoint +family (7770 frank-dragoon, 7771 frank-zealot) — and the engine behind it keeps +8880, addressed as `http://host.containers.internal:8880`, the gateway name a +bridge container uses for the host (`scripts/frank-tts.env`). ## Voice-creation workflow (the "from sample files" answer) @@ -284,6 +374,10 @@ keep clear of the fleet's LLM ports 7770/7771. https://huggingface.co/openbmb/VoxCPM2 - llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni · https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md +- qwentts.cpp (arm A): https://github.com/ServeurpersoCom/qwentts.cpp at commit + `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` · weights + https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF (byte counts against the + Hub API, 2026-10-18) - mainline `llama-tts`: https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md · Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)