dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(tts): record the arm A engine decision and the loopback-publish law

Browse Source 
Frank heard every audition pair and passed it, so likeness stopped being the
tiebreak and the engine was picked on the live-output criterion: arm A
qwentts.cpp 5916bf7, resident tts-server, 1.7 B talker, llama-tts-cli kept as
the rollback. Records the discriminators (first audio byte, warm RTF, spread),
the three consequences that are configuration (boot-warm every voice, no cpuset
pin, WAV-only registration), the two follow-ups left open on purpose, and the
corrected addressing law: a host-loopback publish is unreachable from a bridge
container, so the pair shares one quadlet network.
This commit is contained in:
Frank committed 2026-10-18 16:51:01 -07:00
1 parent 0e7936f94e
commit dbc2c67bcb
2 files changed
+100 -10
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md
+55 -10
View File
@@ -1,7 +1,7 @@
---
type: research
title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts, the full local AI audio stack, and a recommended homelab deployment.
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts and the arm A decision, the full local AI audio stack, and a recommended homelab deployment.
tags:
- research
- tts
@@ -53,9 +53,10 @@ greenfield: every fleet endpoint declares `input: [text, image]`
`/v1/audio/speech` façade that resolves a voice *by name* and never shows a
path. Nothing is deployed — no `frank-tts` unit and no container exist on the
host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet
TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine behind it is measured, not
guessed** (see *Measured CPU engine A/B* below); the pick waits on the
operator's ears.
TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine is chosen: arm A**
`qwentts.cpp` **resident `tts-server`, 1.7 B talker** — measured in
*Measured CPU engine A/B* below and picked on the live-output criterion in
[Decision: arm A for live output](#decision-arm-a-for-live-output-2026-10-18).
## Can Kokoro learn a new voice?
@@ -284,8 +285,11 @@ uncloned 0.9228 — but **two different speakers score 0.9497** on the same
metric. The whole clone effect lives in a 0.04-wide band and the
ICL-over-embedding gap is inside noise, so the four audition clips
(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3`
mainline partner, guide in `LISTEN-engine.md`) are the instrument, and **the
engine pick stays open until the operator has heard them.** The cosine
mainline partner, guide in `LISTEN-engine.md`) are the instrument. **The
operator heard them on 2026-10-18 and passed every clone**, which removed the
last axis measurement could not reach; the engine was then picked on
live-output grounds ([Decision](#decision-arm-a-for-live-output-2026-10-18)).
The cosine
extractor is still worth keeping: it is engine-neutral (it scores mainline
output through the same encoder) and it separates an uncloned voice cleanly.
@@ -298,6 +302,41 @@ endpoint and no audio input, so the crops were transcribed with
suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into
the pack manifest instead of transcribing on the host.
## Decision: arm A for live output (2026-10-18)
Every audition pair passed, so likeness stopped being the tiebreak. The
question became **which engine is better for live output** — the harness
speaking a reply, an alert read aloud, a click-to-speak — and on that criterion
RTF only has to stay under 1.0 with margin: what separates the arms is **time to
first audio** and the **spread**, because an arm whose RTF can cross 1.0 stalls
a live stream mid-sentence.
| | arm A · `qwentts.cpp` `tts-server` | arm B · mainline `llama-tts` |
|---|---|---|
| first audio byte | 86–451 ms | model load, per request |
| warm RTF | 0.55–0.67 | 0.93–0.99 |
| run spread | ±0.01 | ±0.25 |
| HTTP | serves `/v1/audio/speech` | none — façade CLI only |
| RAM | 5.7 GiB resident | per-request, freed after |
**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine`
container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.**
Three consequences are configuration, not opinion: the unit **boot-warms every
voice** (5.5 s cold first byte otherwise), it **does not pin a cpuset** casually
(threads are `hardware_concurrency()/2`, no flag), and every pack registers with
the **WAV** because the engine rejects a lone embedding.
Two follow-ups stay open on purpose, and neither blocks the deploy:
- **Talker size.** The 0.6 B arm measured 0.44 RTF at 3.4 GiB and has never
been auditioned — every clip above is 1.7 B. It is the lever for a CPU
co-tenancy problem, not a preemptive saving.
- **ICL vs embedding-only.** Both passed the ear test. Embedding-only is the
shipped default: 0.57–0.63 vs 0.67 warm, and it needs no `ref_text`, which
this fleet cannot produce honestly (the ASR crops came back suspect). Packs
still carry `.spk` + `.rvq`, so ICL remains a per-voice opt-in for batch work
like the podcast render, where total time matters more than first byte.
## Recommended deployment (Dankclaw)
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
@@ -309,16 +348,22 @@ neither a unit nor a container on the host, and nothing listens on 7772 or
| Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---|
| TTS engine | arm A `qwentts.cpp` `tts-server` on the Qwen3-TTS GGUFs, or arm B mainline `llama-tts`; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself; arm B has no HTTP path and runs through the façade's CLI backend |
| TTS engine | **arm A `qwentts.cpp` `tts-server`** on the Qwen3-TTS GGUFs (decided 2026-10-18, above); arm B mainline `llama-tts` stays in the repo as the `llama-tts-cli` rollback; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 (container-internal) | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself and boot-warms each voice; arm B has no HTTP path and runs through the façade's CLI backend |
| frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices/<name>/reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
7772 is frank-tts's host port — the next slot in the fleet's model-endpoint
family (7770 frank-dragoon, 7771 frank-zealot) — and the engine behind it keeps
8880, addressed as `http://host.containers.internal:8880`, the gateway name a
bridge container uses for the host (`scripts/frank-tts.env`).
family (7770 frank-dragoon, 7771 frank-zealot). The engine behind it keeps 8880,
but **not as a host-loopback publish**: netavark DNATs published ports only, and
a bridge container cannot reach a `127.0.0.1` bind on the host — that is the
lesson that 502'd hermes (`AGENTS.md`). `host.containers.internal`
(169.254.1.2) is the gateway, so it reaches a host service only when that
service binds a routable address; `scripts/frank-tts.env` names it as the
engine's address, which holds for a `0.0.0.0` bind and not for a loopback one.
The deployed pair therefore shares one quadlet network and the façade addresses
the engine by container name.
## Voice-creation workflow (the "from sample files" answer)