dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(tts): record the measured CPU engine A/B behind frank-tts

Browse Source 
Both candidate :8880 engines were measured on Dankclaw's CPU (no GPU,
no container, nothing deployed). arm A qwentts.cpp 5916bf7 resident at
RTF 0.55-0.57 warm / 0.67 ICL / 1.65 cold; arm B mainline llama-tts at
0.93-0.99 with a +-0.25 spread. Records the four configuration laws the
trial produced (boot-warm every voice, .spk+.rvq replay is byte-identical,
streaming != buffered, /v1/audio/voices rejects a lone spk_b64), the
likeness measurement's resolution limit, and re-keys the stale
8890/voiceshim shape on that page to frank-tts on 7772. The engine pick
stays open: the likeness metric cannot rank clones, so the audition clips
decide.
This commit is contained in:
Frank
2026-10-18 16:19:51 -07:00
parent f206ddfde5
commit 0e7936f94e
1 changed files with 119 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
+119 -25
View File
@@ -1,7 +1,7 @@
---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts, the full local AI audio stack, and a recommended homelab deployment.
tags:
- research
- tts
@@ -10,7 +10,7 @@ tags:
- stt
- vllm-omni
- kokoro
timestamp: 2026-10-18T00:00:00Z
timestamp: 2026-10-18T16:05:00Z
---
# Local TTS and Voice Cloning
@@ -47,13 +47,15 @@ greenfield: every fleet endpoint declares `input: [text, image]`
local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
#2 rules out every GPU option (the 3090 has carried frank-zealot since
2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF
engine on **8880** behind a voice-registry shim on **8890**
(`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI
`/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on
2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or
8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality
number has been measured for this shape**, so this page states none.
2026-10-17; both R9700s are frank-archon's). The shape is a TTS engine on
**8880** behind **frank-tts**, the fleet's speech endpoint on **7772**
(`scripts/frank-tts.env`, `dankclaw/voiceservice/`): one OpenAI
`/v1/audio/speech` façade that resolves a voice *by name* and never shows a
path. Nothing is deployed — no `frank-tts` unit and no container exist on the
host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet
TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine behind it is measured, not
guessed** (see *Measured CPU engine A/B* below); the pick waits on the
operator's ears.
## Can Kokoro learn a new voice?
@@ -133,11 +135,11 @@ accepts only its built-in voice list, so a custom pack needs a patch.
Paths for a Frank-cloned voice (2026-10-18):
1. **Local server + speech role, no patch.** Register the local engine (or the
8890 voice shim) as a provider model on the `openai-speech` API and point
`modelRoles.speech` at it; the shim turns a voice *name* into a reference
clip and forwards it as base64 `reference_audio`
(`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18.
1. **Local server + speech role, no patch.** Register frank-tts (port 7772) as
a provider model on the `openai-speech` API and point `modelRoles.speech` at
it; the façade turns a voice *name* into that pack's reference clip and
forwards it as base64 `reference_audio` (`scripts/frank-tts.env`,
`TTS_BACKEND=omni-http`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
tool. Free (≤30 voices), cloud, US-only.
@@ -206,29 +208,117 @@ The CPU shape does not need a Python server at all:
`VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
`tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.
Nothing has been measured on this fleet for any of these engines — no RTF, no
latency, no MOS. That absence is the finding, not an omission.
Arm B (`llama-tts`) and arm A (`qwentts.cpp`) have both been measured on this
box — RTF, first-audio latency and RSS are in the next section. The other
servers in the table above (vLLM-Omni, Kokoro-FastAPI, CosyVoice, F5-TTS,
GPT-SoVITS) remain unmeasured here; that absence is still the finding.
## Measured CPU engine A/B (2026-10-18)
Both candidate upstreams for frank-tts were measured on Dankclaw's CPU — 8
threads, no GPU touched, three runs per arm, seed 42, identical text, against
the fleet's own audition voice packs.
- **Arm A — [`ServeurpersoCom/qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp)**
at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (2026-10-16), weights
from [`Serveurperso/Qwen3-TTS-GGUF`](https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF):
`qwen-talker-1.7b-base-Q4_K_M.gguf` (1,219,245,248 B),
`qwen-talker-0.6b-base-Q4_K_M.gguf` (628,905,056 B),
`qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B) — byte counts verified
against the Hub API. Builds with zero compiler warnings; `GGML_BLAS=OFF` is
forced on this host (only `libopenblas.so.0` exists, no dev headers, so the
vendor `buildcpu.sh` cannot configure).
- **Arm B — mainline `llama-tts`** with `--tts-speaker-file`, i.e. the
`llama-tts-cli` backend `scripts/frank-tts.env` already implements.
RTF is **seconds of compute per second of audio** (lower is better; below 1 is
faster than real time).
| Arm | RTF | Run spread | Peak RSS |
|---|---|---|---|
| A · 0.6 B talker, CLI | 0.44 | ±0.002 | 3.4 GiB |
| A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB |
| A · 1.7 B, CLI, embedding-only clone | 0.63 | ±0.02 | n/m |
| A · 1.7 B, CLI, ICL clone | 0.92 | ±0.01 | n/m |
| A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB |
| A · same, ICL voice warm / **cold (first request)** | 0.67 / **1.65** | one-shot | 5.7 GiB |
| B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m |
**Read RTF only within an arm.** Conditioning changes clip length — the same
seed yields 18.00 s of audio embedding-only and 15.12 s under ICL — so the
denominator is the arm's own audio duration.
Arm A's resident server is ~1.7× faster than arm B and about 40× tighter in
run-to-run spread, and it already serves the HTTP shape frank-tts's façade
speaks: `/health`, `/v1/models`, `/v1/audio/voices` (GET/POST/DELETE),
`/v1/audio/speech` (`wav`/`pcm`), chunked streaming with the first audio byte
at 86–451 ms and a server-side time-to-first-audio of 94–112 ms. Arm B has no
HTTP path at all — that is what the façade's CLI backend exists for.
Four laws came out of the trial, each worth a line of configuration:
- **Boot-warm every voice.** The first request for a registered voice decodes
the whole reference to snapshot codec state: 6.2–8.0 s, first audio byte at
5,530 ms. Later requests restore that state in 0.5 ms. An engine unit must
run one throwaway synthesis per voice at start, or the first request of the
day pays five seconds.
- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference
WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T,
11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a
**byte-identical** WAV and saves 1.4 s per request. That pair is the
engine-artifact term the voice-pack law in `scripts/frank-tts.env` reserves a
slot for — arm A pins it as `.spk` + `.rvq` beside `reference.wav`.
- **Streaming is not bit-identical to buffered**, whatever the upstream README
claims: same text, seed and voice, 92.2 % of samples differ (peak delta
1139/32768, −41.6 dB RMS — inaudible). A fixed seed does not pin output
across `response_format`, so never mix the two paths in a cache or a golden
fixture.
- **A voice cannot be registered from an embedding alone**: `POST
/v1/audio/voices` accepts `wav_b64` or **both** `spk_b64` and `rvq_b64`
(`src/tts-server.h:480`), so an embedding-only voice has to be sent as the
WAV, or the lone-`spk` path has to be upstreamed.
**Likeness is not measurable at this resolution.** x-vector cosine against each
speaker's own reference: ICL 0.9886 / 0.9901, embedding-only 0.9844 / 0.9812,
uncloned 0.9228 — but **two different speakers score 0.9497** on the same
metric. The whole clone effect lives in a 0.04-wide band and the
ICL-over-embedding gap is inside noise, so the four audition clips
(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3`
mainline partner, guide in `LISTEN-engine.md`) are the instrument, and **the
engine pick stays open until the operator has heard them.** The cosine
extractor is still worth keeping: it is engine-neutral (it scores mainline
output through the same encoder) and it separates an uncloned voice cleanly.
Two operational details. The engine takes 8 threads and exposes **no `-t`
flag** — `src/backend.h` uses `hardware_concurrency()/2` and prints the choice
at load, so pinning a cpuset on the quadlet changes that number implicitly. And
ICL needs a reference transcript that did not exist: the fleet has no ASR
endpoint and no audio input, so the crops were transcribed with
`vosk-small-en-us` inside the work dir, and two of three transcripts came back
suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into
the pack manifest instead of transcribing on the host.
## Recommended deployment (Dankclaw)
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus a thin
voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl
--user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or
8890.
so no card may be asked to co-host TTS. The shape is one engine plus one
façade, and **nothing here is deployed as of 2026-10-18**: `frank-tts` has
neither a unit nor a container on the host, and nothing listens on 7772 or
8880.
| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot |
| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| TTS engine | arm A `qwentts.cpp` `tts-server` on the Qwen3-TTS GGUFs, or arm B mainline `llama-tts`; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself; arm B has no HTTP path and runs through the façade's CLI backend |
| frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices/<name>/reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM=
http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT
keep clear of the fleet's LLM ports 7770/7771.
7772 is frank-tts's host port — the next slot in the fleet's model-endpoint
family (7770 frank-dragoon, 7771 frank-zealot) — and the engine behind it keeps
8880, addressed as `http://host.containers.internal:8880`, the gateway name a
bridge container uses for the host (`scripts/frank-tts.env`).
## Voice-creation workflow (the "from sample files" answer)
@@ -284,6 +374,10 @@ keep clear of the fleet's LLM ports 7770/7771.
https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- qwentts.cpp (arm A): https://github.com/ServeurpersoCom/qwentts.cpp at commit
`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` · weights
https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF (byte counts against the
Hub API, 2026-10-18)
- mainline `llama-tts`:
https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)