diff --git a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md index f04bd7b..836b68f 100644 --- a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md +++ b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md @@ -1,7 +1,7 @@ --- type: research title: Local TTS and Voice Cloning -description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts, the full local AI audio stack, and a recommended homelab deployment. +description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts and the arm A decision, the full local AI audio stack, and a recommended homelab deployment. tags: - research - tts @@ -53,9 +53,10 @@ greenfield: every fleet endpoint declares `input: [text, image]` `/v1/audio/speech` façade that resolves a voice *by name* and never shows a path. Nothing is deployed — no `frank-tts` unit and no container exist on the host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet - TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine behind it is measured, not - guessed** (see *Measured CPU engine A/B* below); the pick waits on the - operator's ears. + TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine is chosen: arm A** + `qwentts.cpp` **resident `tts-server`, 1.7 B talker** — measured in + *Measured CPU engine A/B* below and picked on the live-output criterion in + [Decision: arm A for live output](#decision-arm-a-for-live-output-2026-10-18). ## Can Kokoro learn a new voice? @@ -284,8 +285,11 @@ uncloned 0.9228 — but **two different speakers score 0.9497** on the same metric. The whole clone effect lives in a 0.04-wide band and the ICL-over-embedding gap is inside noise, so the four audition clips (`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3` -mainline partner, guide in `LISTEN-engine.md`) are the instrument, and **the -engine pick stays open until the operator has heard them.** The cosine +mainline partner, guide in `LISTEN-engine.md`) are the instrument. **The +operator heard them on 2026-10-18 and passed every clone**, which removed the +last axis measurement could not reach; the engine was then picked on +live-output grounds ([Decision](#decision-arm-a-for-live-output-2026-10-18)). +The cosine extractor is still worth keeping: it is engine-neutral (it scores mainline output through the same encoder) and it separates an uncloned voice cleanly. @@ -298,6 +302,41 @@ endpoint and no audio input, so the crops were transcribed with suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into the pack manifest instead of transcribing on the host. +## Decision: arm A for live output (2026-10-18) + +Every audition pair passed, so likeness stopped being the tiebreak. The +question became **which engine is better for live output** — the harness +speaking a reply, an alert read aloud, a click-to-speak — and on that criterion +RTF only has to stay under 1.0 with margin: what separates the arms is **time to +first audio** and the **spread**, because an arm whose RTF can cross 1.0 stalls +a live stream mid-sentence. + +| | arm A · `qwentts.cpp` `tts-server` | arm B · mainline `llama-tts` | +|---|---|---| +| first audio byte | 86–451 ms | model load, per request | +| warm RTF | 0.55–0.67 | 0.93–0.99 | +| run spread | ±0.01 | ±0.25 | +| HTTP | serves `/v1/audio/speech` | none — façade CLI only | +| RAM | 5.7 GiB resident | per-request, freed after | + +**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine` +container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.** +Three consequences are configuration, not opinion: the unit **boot-warms every +voice** (5.5 s cold first byte otherwise), it **does not pin a cpuset** casually +(threads are `hardware_concurrency()/2`, no flag), and every pack registers with +the **WAV** because the engine rejects a lone embedding. + +Two follow-ups stay open on purpose, and neither blocks the deploy: + +- **Talker size.** The 0.6 B arm measured 0.44 RTF at 3.4 GiB and has never + been auditioned — every clip above is 1.7 B. It is the lever for a CPU + co-tenancy problem, not a preemptive saving. +- **ICL vs embedding-only.** Both passed the ear test. Embedding-only is the + shipped default: 0.57–0.63 vs 0.67 warm, and it needs no `ref_text`, which + this fleet cannot produce honestly (the ASR crops came back suspect). Packs + still carry `.spk` + `.rvq`, so ICL remains a per-voice opt-in for batch work + like the podcast render, where total time matters more than first byte. + ## Recommended deployment (Dankclaw) As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090 @@ -309,16 +348,22 @@ neither a unit nor a container on the host, and nothing listens on 7772 or | Service | Image / install | Port | CPU | Notes | |---|---|---|---|---| -| TTS engine | arm A `qwentts.cpp` `tts-server` on the Qwen3-TTS GGUFs, or arm B mainline `llama-tts`; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself; arm B has no HTTP path and runs through the façade's CLI backend | +| TTS engine | **arm A `qwentts.cpp` `tts-server`** on the Qwen3-TTS GGUFs (decided 2026-10-18, above); arm B mainline `llama-tts` stays in the repo as the `llama-tts-cli` rollback; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 (container-internal) | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself and boot-warms each voice; arm B has no HTTP path and runs through the façade's CLI backend | | frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices//reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) | | STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) | | STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT | | VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B | 7772 is frank-tts's host port — the next slot in the fleet's model-endpoint -family (7770 frank-dragoon, 7771 frank-zealot) — and the engine behind it keeps -8880, addressed as `http://host.containers.internal:8880`, the gateway name a -bridge container uses for the host (`scripts/frank-tts.env`). +family (7770 frank-dragoon, 7771 frank-zealot). The engine behind it keeps 8880, +but **not as a host-loopback publish**: netavark DNATs published ports only, and +a bridge container cannot reach a `127.0.0.1` bind on the host — that is the +lesson that 502'd hermes (`AGENTS.md`). `host.containers.internal` +(169.254.1.2) is the gateway, so it reaches a host service only when that +service binds a routable address; `scripts/frank-tts.env` names it as the +engine's address, which holds for a `0.0.0.0` bind and not for a loopback one. +The deployed pair therefore shares one quadlet network and the façade addresses +the engine by container name. ## Voice-creation workflow (the "from sample files" answer) diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index b397e67..cb98a01 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2311,3 +2311,48 @@ every restart since the cutover. Kokoro-enum-only — so "the agent talks in my voice" needs that provider row or a `pi.registerProvider` extension, decided in phase 2. +## 2026-10-18 — TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion + +- Frank listened to the four audition pairs (`qwentts-*.mp3` against their + `armB-*.mp3` mainline partners) and **passed every clone**. That deleted the + only axis the A/B could not measure, and the question was reframed: **which + engine is better for live output**. On that criterion RTF only has to stay + under 1.0 with margin — the discriminators are time-to-first-audio and + run-to-run spread, because an arm whose RTF can cross 1.0 stalls a live + stream mid-sentence. +- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`, 1.7 B + talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on 8880.** + The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env` stays in the + repo as the rollback. The numbers that carried it: first audio byte 86–451 ms + against a per-request model load, warm RTF 0.55–0.67 against 0.93–0.99, spread + ±0.01 against ±0.25, and a native `/v1/audio/speech` against no HTTP path at + all. Three consequences are configuration, not preference: boot-warm every + voice, do not pin a cpuset (threads are `hardware_concurrency()/2`), register + every pack with its WAV (a lone embedding is rejected). +- Two follow-ups deliberately **not** decided: the **0.6 B talker** (0.44 RTF, + 3.4 GiB, never auditioned — every clip is 1.7 B) and **ICL vs + embedding-only** conditioning. Embedding-only is the shipped default because + it is cheaper and needs no `ref_text`, which this fleet cannot transcribe + honestly; `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice + opt-in for batch work where total time beats first byte. +- **Addressing law corrected** in + [[Local TTS and Voice Cloning]]: the pair may not meet at a host-loopback + publish. netavark DNATs published ports only and a bridge container cannot + reach a host `127.0.0.1` bind — the lesson that 502'd hermes — so + `host.containers.internal:8880`, which `scripts/frank-tts.env` documents, is + correct only for a routable bind. The deployed pair shares one quadlet network + and the façade addresses the engine by container name. +- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`, 8 commits, 403 tests, + mergeable) is open: open-notebook + SurrealDB quadlets on the private bridge + `10.89.4.0/24`, API published on **5056** because seerr holds + `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT` moved 600 → **660** to sit one + round-trip outside frank-tts's own `SHIM_TIMEOUT`. Stated in its body: it is + **not runnable end to end** — nothing listens on 7772 until the voice stack + lands, and the episode destination (`/home/dank/media/podcasts`) is interim + pending Gitea #7. +- Still in flight, not in this branch: `feat/frank-tts-engine` — the arm A + image build, the quadlet pair on one network, the boot-warm step, deployer / + planner / `test-changed.sh` routing, homepage and docs. Whoever merges second + among the open branches resolves the overlap: `log.md`, `index.md`, + `deploy-remote.sh`, `deploy-plan.py`, `test-changed.sh`. +