dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into feat/voice-pack-builder

Browse Source
This commit is contained in:
Frank
2026-10-18 17:18:31 -07:00
parent 042003a969 e13f875c94
commit 3824d03ac5
2 changed files with 210 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
+164 -25
View File
@@ -1,7 +1,7 @@
---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment.
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts and the arm A decision, the full local AI audio stack, and a recommended homelab deployment.
tags:
- research
- tts
@@ -10,7 +10,7 @@ tags:
- stt
- vllm-omni
- kokoro
timestamp: 2026-10-18T00:00:00Z
timestamp: 2026-10-18T16:05:00Z
---
# Local TTS and Voice Cloning
@@ -47,13 +47,16 @@ greenfield: every fleet endpoint declares `input: [text, image]`
local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
#2 rules out every GPU option (the 3090 has carried frank-zealot since
2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF
engine on **8880** behind a voice-registry shim on **8890**
(`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI
`/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on
2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or
8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality
number has been measured for this shape**, so this page states none.
2026-10-17; both R9700s are frank-archon's). The shape is a TTS engine on
**8880** behind **frank-tts**, the fleet's speech endpoint on **7772**
(`scripts/frank-tts.env`, `dankclaw/voiceservice/`): one OpenAI
`/v1/audio/speech` façade that resolves a voice *by name* and never shows a
path. Nothing is deployed — no `frank-tts` unit and no container exist on the
host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet
TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine is chosen: arm A**
`qwentts.cpp` **resident `tts-server`, 1.7 B talker** — measured in
*Measured CPU engine A/B* below and picked on the live-output criterion in
[Decision: arm A for live output](#decision-arm-a-for-live-output-2026-10-18).
## Can Kokoro learn a new voice?
@@ -133,11 +136,11 @@ accepts only its built-in voice list, so a custom pack needs a patch.
Paths for a Frank-cloned voice (2026-10-18):
1. **Local server + speech role, no patch.** Register the local engine (or the
8890 voice shim) as a provider model on the `openai-speech` API and point
`modelRoles.speech` at it; the shim turns a voice *name* into a reference
clip and forwards it as base64 `reference_audio`
(`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18.
1. **Local server + speech role, no patch.** Register frank-tts (port 7772) as
a provider model on the `openai-speech` API and point `modelRoles.speech` at
it; the façade turns a voice *name* into that pack's reference clip and
forwards it as base64 `reference_audio` (`scripts/frank-tts.env`,
`TTS_BACKEND=omni-http`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
tool. Free (≤30 voices), cloud, US-only.
@@ -206,29 +209,161 @@ The CPU shape does not need a Python server at all:
`VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
`tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.
Nothing has been measured on this fleet for any of these engines — no RTF, no
latency, no MOS. That absence is the finding, not an omission.
Arm B (`llama-tts`) and arm A (`qwentts.cpp`) have both been measured on this
box — RTF, first-audio latency and RSS are in the next section. The other
servers in the table above (vLLM-Omni, Kokoro-FastAPI, CosyVoice, F5-TTS,
GPT-SoVITS) remain unmeasured here; that absence is still the finding.
## Measured CPU engine A/B (2026-10-18)
Both candidate upstreams for frank-tts were measured on Dankclaw's CPU — 8
threads, no GPU touched, three runs per arm, seed 42, identical text, against
the fleet's own audition voice packs.
- **Arm A — [`ServeurpersoCom/qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp)**
at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (2026-10-16), weights
from [`Serveurperso/Qwen3-TTS-GGUF`](https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF):
`qwen-talker-1.7b-base-Q4_K_M.gguf` (1,219,245,248 B),
`qwen-talker-0.6b-base-Q4_K_M.gguf` (628,905,056 B),
`qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B) — byte counts verified
against the Hub API. Builds with zero compiler warnings; `GGML_BLAS=OFF` is
forced on this host (only `libopenblas.so.0` exists, no dev headers, so the
vendor `buildcpu.sh` cannot configure).
- **Arm B — mainline `llama-tts`** with `--tts-speaker-file`, i.e. the
`llama-tts-cli` backend `scripts/frank-tts.env` already implements.
RTF is **seconds of compute per second of audio** (lower is better; below 1 is
faster than real time).
| Arm | RTF | Run spread | Peak RSS |
|---|---|---|---|
| A · 0.6 B talker, CLI | 0.44 | ±0.002 | 3.4 GiB |
| A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB |
| A · 1.7 B, CLI, embedding-only clone | 0.63 | ±0.02 | n/m |
| A · 1.7 B, CLI, ICL clone | 0.92 | ±0.01 | n/m |
| A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB |
| A · same, ICL voice warm / **cold (first request)** | 0.67 / **1.65** | one-shot | 5.7 GiB |
| B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m |
**Read RTF only within an arm.** Conditioning changes clip length — the same
seed yields 18.00 s of audio embedding-only and 15.12 s under ICL — so the
denominator is the arm's own audio duration.
Arm A's resident server is ~1.7× faster than arm B and about 40× tighter in
run-to-run spread, and it already serves the HTTP shape frank-tts's façade
speaks: `/health`, `/v1/models`, `/v1/audio/voices` (GET/POST/DELETE),
`/v1/audio/speech` (`wav`/`pcm`), chunked streaming with the first audio byte
at 86–451 ms and a server-side time-to-first-audio of 94–112 ms. Arm B has no
HTTP path at all — that is what the façade's CLI backend exists for.
Four laws came out of the trial, each worth a line of configuration:
- **Boot-warm every voice.** The first request for a registered voice decodes
the whole reference to snapshot codec state: 6.2–8.0 s, first audio byte at
5,530 ms. Later requests restore that state in 0.5 ms. An engine unit must
run one throwaway synthesis per voice at start, or the first request of the
day pays five seconds.
- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference
WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T,
11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a
**byte-identical** WAV and saves 1.4 s per request. That pair is the
engine-artifact term the voice-pack law in `scripts/frank-tts.env` reserves a
slot for — arm A pins it as `.spk` + `.rvq` beside `reference.wav`.
- **Streaming is not bit-identical to buffered**, whatever the upstream README
claims: same text, seed and voice, 92.2 % of samples differ (peak delta
1139/32768, −41.6 dB RMS — inaudible). A fixed seed does not pin output
across `response_format`, so never mix the two paths in a cache or a golden
fixture.
- **A voice cannot be registered from an embedding alone**: `POST
/v1/audio/voices` accepts `wav_b64` or **both** `spk_b64` and `rvq_b64`
(`src/tts-server.h:480`), so an embedding-only voice has to be sent as the
WAV, or the lone-`spk` path has to be upstreamed.
**Likeness is not measurable at this resolution.** x-vector cosine against each
speaker's own reference: ICL 0.9886 / 0.9901, embedding-only 0.9844 / 0.9812,
uncloned 0.9228 — but **two different speakers score 0.9497** on the same
metric. The whole clone effect lives in a 0.04-wide band and the
ICL-over-embedding gap is inside noise, so the four audition clips
(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3`
mainline partner, guide in `LISTEN-engine.md`) are the instrument. **The
operator heard them on 2026-10-18 and passed every clone**, which removed the
last axis measurement could not reach; the engine was then picked on
live-output grounds ([Decision](#decision-arm-a-for-live-output-2026-10-18)).
The cosine
extractor is still worth keeping: it is engine-neutral (it scores mainline
output through the same encoder) and it separates an uncloned voice cleanly.
Two operational details. The engine takes 8 threads and exposes **no `-t`
flag** — `src/backend.h` uses `hardware_concurrency()/2` and prints the choice
at load, so pinning a cpuset on the quadlet changes that number implicitly. And
ICL needs a reference transcript that did not exist: the fleet has no ASR
endpoint and no audio input, so the crops were transcribed with
`vosk-small-en-us` inside the work dir, and two of three transcripts came back
suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into
the pack manifest instead of transcribing on the host.
## Decision: arm A for live output (2026-10-18)
Every audition pair passed, so likeness stopped being the tiebreak. The
question became **which engine is better for live output** — the harness
speaking a reply, an alert read aloud, a click-to-speak — and on that criterion
RTF only has to stay under 1.0 with margin: what separates the arms is **time to
first audio** and the **spread**, because an arm whose RTF can cross 1.0 stalls
a live stream mid-sentence.
| | arm A · `qwentts.cpp` `tts-server` | arm B · mainline `llama-tts` |
|---|---|---|
| first audio byte | 86–451 ms | model load, per request |
| warm RTF | 0.55–0.67 | 0.93–0.99 |
| run spread | ±0.01 | ±0.25 |
| HTTP | serves `/v1/audio/speech` | none — façade CLI only |
| RAM | 5.7 GiB resident | per-request, freed after |
**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine`
container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.**
Three consequences are configuration, not opinion: the unit **boot-warms every
voice** (5.5 s cold first byte otherwise), it **does not pin a cpuset** casually
(threads are `hardware_concurrency()/2`, no flag), and every pack registers with
the **WAV** because the engine rejects a lone embedding.
Two follow-ups stay open on purpose, and neither blocks the deploy:
- **Talker size.** The 0.6 B arm measured 0.44 RTF at 3.4 GiB and has never
been auditioned — every clip above is 1.7 B. It is the lever for a CPU
co-tenancy problem, not a preemptive saving.
- **ICL vs embedding-only.** Both passed the ear test. Embedding-only is the
shipped default: 0.57–0.63 vs 0.67 warm, and it needs no `ref_text`, which
this fleet cannot produce honestly (the ASR crops came back suspect). Packs
still carry `.spk` + `.rvq`, so ICL remains a per-voice opt-in for batch work
like the podcast render, where total time matters more than first byte.
## Recommended deployment (Dankclaw)
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus a thin
voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl
--user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or
8890.
so no card may be asked to co-host TTS. The shape is one engine plus one
façade, and **nothing here is deployed as of 2026-10-18**: `frank-tts` has
neither a unit nor a container on the host, and nothing listens on 7772 or
8880.
| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot |
| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| TTS engine | **arm A `qwentts.cpp` `tts-server`** on the Qwen3-TTS GGUFs (decided 2026-10-18, above); arm B mainline `llama-tts` stays in the repo as the `llama-tts-cli` rollback; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 (container-internal) | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself and boot-warms each voice; arm B has no HTTP path and runs through the façade's CLI backend |
| frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices/<name>/reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM=
http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT
keep clear of the fleet's LLM ports 7770/7771.
7772 is frank-tts's host port — the next slot in the fleet's model-endpoint
family (7770 frank-dragoon, 7771 frank-zealot). The engine behind it keeps 8880,
but **not as a host-loopback publish**: netavark DNATs published ports only, and
a bridge container cannot reach a `127.0.0.1` bind on the host — that is the
lesson that 502'd hermes (`AGENTS.md`). `host.containers.internal`
(169.254.1.2) is the gateway, so it reaches a host service only when that
service binds a routable address; `scripts/frank-tts.env` names it as the
engine's address, which holds for a `0.0.0.0` bind and not for a loopback one.
The deployed pair therefore shares one quadlet network and the façade addresses
the engine by container name.
## Voice-creation workflow (the "from sample files" answer)
@@ -284,6 +419,10 @@ keep clear of the fleet's LLM ports 7770/7771.
https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- qwentts.cpp (arm A): https://github.com/ServeurpersoCom/qwentts.cpp at commit
`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` · weights
https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF (byte counts against the
Hub API, 2026-10-18)
- mainline `llama-tts`:
https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)
serverAppConfigs-docs/log.md
+46
View File
@@ -2311,6 +2311,52 @@ every restart since the cutover.
Kokoro-enum-only — so "the agent talks in my voice" needs that provider row or
a `pi.registerProvider` extension, decided in phase 2.
## 2026-10-18 — TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion
- Frank listened to the four audition pairs (`qwentts-*.mp3` against their
`armB-*.mp3` mainline partners) and **passed every clone**. That deleted the
only axis the A/B could not measure, and the question was reframed: **which
engine is better for live output**. On that criterion RTF only has to stay
under 1.0 with margin — the discriminators are time-to-first-audio and
run-to-run spread, because an arm whose RTF can cross 1.0 stalls a live
stream mid-sentence.
- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`, 1.7 B
talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on 8880.**
The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env` stays in the
repo as the rollback. The numbers that carried it: first audio byte 86–451 ms
against a per-request model load, warm RTF 0.55–0.67 against 0.93–0.99, spread
±0.01 against ±0.25, and a native `/v1/audio/speech` against no HTTP path at
all. Three consequences are configuration, not preference: boot-warm every
voice, do not pin a cpuset (threads are `hardware_concurrency()/2`), register
every pack with its WAV (a lone embedding is rejected).
- Two follow-ups deliberately **not** decided: the **0.6 B talker** (0.44 RTF,
3.4 GiB, never auditioned — every clip is 1.7 B) and **ICL vs
embedding-only** conditioning. Embedding-only is the shipped default because
it is cheaper and needs no `ref_text`, which this fleet cannot transcribe
honestly; `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice
opt-in for batch work where total time beats first byte.
- **Addressing law corrected** in
[[Local TTS and Voice Cloning]]: the pair may not meet at a host-loopback
publish. netavark DNATs published ports only and a bridge container cannot
reach a host `127.0.0.1` bind — the lesson that 502'd hermes — so
`host.containers.internal:8880`, which `scripts/frank-tts.env` documents, is
correct only for a routable bind. The deployed pair shares one quadlet network
and the façade addresses the engine by container name.
- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`, 8 commits, 403 tests) is
**merged** as `1eb8b21` (Frank's OK, given in session): open-notebook +
SurrealDB quadlets on the private bridge `10.89.4.0/24`, the API published on
**5056** because seerr holds `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT`
moved 600 → **660** to sit one round-trip outside frank-tts's own
`SHIM_TIMEOUT`. Merged is not runnable: nothing listens on 7772 until the
voice stack lands, and the episode destination (`/home/dank/media/podcasts`)
stays interim pending Gitea #7.
- Still in flight, not in this branch: `feat/frank-tts-engine` — the arm A image
build, the quadlet pair on one network, the boot-warm step, deployer /
planner / `test-changed.sh` routing, homepage and docs. This branch is also
where the two-branch overlap got resolved: merging `main` here conflicted on
`log.md` alone (both branches appended an entry) and took `index.md`,
`deploy-remote.sh`, `deploy-plan.py` and `test-changed.sh` clean.
## 2026-10-18 — open-notebook: the podcast generator's config lands (nothing deployed)
- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — notes