docs(tts): record the measured CPU engine A/B behind frank-tts #10

Merged
dank merged 4 commits from docs/tts-engine-trials into main 2026-10-18 17:15:00 -07:00
Owner

What this changes

Two documentation files. No code, no unit, no deploy, no host touched.

serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md — the page said "Nothing has been measured on this fleet for any of these engines — no RTF, no latency, no MOS." That is no longer true, and the page also still described the retired voiceshim on 8890 with scripts/voiceshim.env (a file that no longer exists). Both are corrected:

  • New section Measured CPU engine A/B (2026-10-18): arms, provenance (commit + Hub byte counts), an RTF table, and the four configuration laws the trial produced.
  • Fleet-plan summary, the OMP integration path, and the deployment table re-keyed to frank-tts on 7772 (scripts/frank-tts.env, dankclaw/voiceservice/), with the still-true fact that nothing is deployed — verified today: no unit, no container, nothing listening on 7772 or 8880.
  • Citations: ServeurpersoCom/qwentts.cpp @ 5916bf7c… and Serveurperso/Qwen3-TTS-GGUF.

serverAppConfigs-docs/log.md — worklog entry for the trial, plus the tracker bookkeeping (episode-one source staged, #8 opened, #5 rewritten to a Telegram bot and then parked in #9) and an agent-tool hazard record.

The measurement

CPU only, Dankclaw, 8 threads, seed 42, three runs per arm, no GPU touched, MemAvailable 21 GiB before and after. RTF = seconds of compute per second of audio.

Arm RTF Run spread Peak RSS
A · qwentts.cpp 0.6 B, CLI 0.44 ±0.002 3.4 GiB
A · 1.7 B, CLI, uncloned 0.57–0.58 ±0.006 4.7 GiB
A · 1.7 B, resident tts-server, warm 0.55–0.57 ±0.01 5.7 GiB
A · same, ICL voice warm / cold 0.67 / 1.65 one-shot 5.7 GiB
B · mainline llama-tts 0.93–0.99 ±0.25 n/m

RTF is comparable only within an arm — conditioning changes clip length (same seed: 18.00 s embedding-only vs 15.12 s ICL).

Four laws now written down, because each one costs a line of configuration:

  1. Boot-warm every voice — the first request per voice snapshots codec state for 6.2–8.0 s (first audio byte 5,530 ms); later ones restore in 0.5 ms.
  2. .spk + .rvq replay is byte-identical to re-reading the reference WAV and saves 1.4 s/request — that pair is the engine-artifact term the voice-pack law reserves a slot for.
  3. Streaming ≠ buffered (92.2 % of samples differ, −41.6 dB RMS) despite the upstream README — a seed does not pin output across response_format, so never mix the two paths in a cache or fixture.
  4. POST /v1/audio/voices rejects a lone spk_b64 — an embedding-only voice cannot be registered from a precomputed x-vector.

What is deliberately NOT decided

The engine pick. x-vector likeness cosine gives ICL 0.9886/0.9901, embedding-only 0.9844/0.9812, uncloned 0.9228 — but two different speakers score 0.9497 on the same metric, so the whole clone effect sits in a 0.04-wide band and the ICL-over-embedding gap is noise. The deciding instrument is the operator's ears on /home/blank/voice-audition/qwentts-*.mp3 vs their armB-*.mp3 partners, so the page states the evidence and leaves the choice open.

Verification

  • ./tests/run.sh → ==> ALL TESTS PASSED (384 tests, wall 65.74 s), run in this branch's worktree after both files were edited.
  • Deployed-state claims re-checked against the host before writing: systemctl/podman show no frank-tts unit or container, and ss -ltnp shows nothing on 7772/8880.
  • Weight byte counts and the engine commit re-read from the host clone (git remote -v, git log -1) and the Hub API.

Update 2026-10-18 — the engine is now decided

Frank heard the four audition pairs and passed every clone, then chose on the criterion "whatever is better for live output" → arm A qwentts.cpp @ 5916bf7, resident tts-server, 1.7 B talker, with TTS_BACKEND=llama-tts-cli kept as the rollback. This branch now carries that decision (new Decision: arm A for live output section), the corrected addressing law (the pair may not meet at a host-loopback publish — host.containers.internal is the gateway, so a 127.0.0.1 bind is unreachable from a bridge container), and the two follow-ups left deliberately open (0.6 B talker, ICL vs embedding-only). Build worktree for this branch: /home/blank/.omp/wt/wt-tts-trial-docs.


cwd: /home/blank/.omp/wt/wt-20261017-181502-6d1fc1f · session: 01a14c93-8870-75d9-9900-9a731559feef

## What this changes Two documentation files. **No code, no unit, no deploy, no host touched.** `serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md` — the page said *"Nothing has been measured on this fleet for any of these engines — no RTF, no latency, no MOS."* That is no longer true, and the page also still described the retired `voiceshim` on **8890** with `scripts/voiceshim.env` (a file that no longer exists). Both are corrected: - New section **Measured CPU engine A/B (2026-10-18)**: arms, provenance (commit + Hub byte counts), an RTF table, and the four configuration laws the trial produced. - Fleet-plan summary, the OMP integration path, and the deployment table re-keyed to **frank-tts on 7772** (`scripts/frank-tts.env`, `dankclaw/voiceservice/`), with the still-true fact that **nothing is deployed** — verified today: no unit, no container, nothing listening on 7772 or 8880. - Citations: `ServeurpersoCom/qwentts.cpp` @ `5916bf7c…` and `Serveurperso/Qwen3-TTS-GGUF`. `serverAppConfigs-docs/log.md` — worklog entry for the trial, plus the tracker bookkeeping (episode-one source staged, #8 opened, #5 rewritten to a Telegram bot and then parked in #9) and an agent-tool hazard record. ## The measurement CPU only, Dankclaw, 8 threads, seed 42, three runs per arm, no GPU touched, `MemAvailable` 21 GiB before and after. RTF = seconds of compute per second of audio. | Arm | RTF | Run spread | Peak RSS | |---|---|---|---| | A · qwentts.cpp 0.6 B, CLI | 0.44 | ±0.002 | 3.4 GiB | | A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB | | A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB | | A · same, ICL voice warm / **cold** | 0.67 / **1.65** | one-shot | 5.7 GiB | | B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m | RTF is comparable **only within an arm** — conditioning changes clip length (same seed: 18.00 s embedding-only vs 15.12 s ICL). Four laws now written down, because each one costs a line of configuration: 1. **Boot-warm every voice** — the first request per voice snapshots codec state for 6.2–8.0 s (first audio byte 5,530 ms); later ones restore in 0.5 ms. 2. **`.spk` + `.rvq` replay is byte-identical** to re-reading the reference WAV and saves 1.4 s/request — that pair is the engine-artifact term the voice-pack law reserves a slot for. 3. **Streaming ≠ buffered** (92.2 % of samples differ, −41.6 dB RMS) despite the upstream README — a seed does not pin output across `response_format`, so never mix the two paths in a cache or fixture. 4. **`POST /v1/audio/voices` rejects a lone `spk_b64`** — an embedding-only voice cannot be registered from a precomputed x-vector. ## What is deliberately NOT decided **The engine pick.** x-vector likeness cosine gives ICL 0.9886/0.9901, embedding-only 0.9844/0.9812, uncloned 0.9228 — but two *different speakers* score 0.9497 on the same metric, so the whole clone effect sits in a 0.04-wide band and the ICL-over-embedding gap is noise. The deciding instrument is the operator's ears on `/home/blank/voice-audition/qwentts-*.mp3` vs their `armB-*.mp3` partners, so the page states the evidence and leaves the choice open. ## Verification - `./tests/run.sh` → **`==> ALL TESTS PASSED`** (384 tests, wall 65.74 s), run in this branch's worktree after both files were edited. - Deployed-state claims re-checked against the host before writing: `systemctl`/`podman` show no `frank-tts` unit or container, and `ss -ltnp` shows nothing on 7772/8880. - Weight byte counts and the engine commit re-read from the host clone (`git remote -v`, `git log -1`) and the Hub API. ## Update 2026-10-18 — the engine is now decided Frank heard the four audition pairs and passed every clone, then chose on the criterion **"whatever is better for live output"** → **arm A `qwentts.cpp` @ `5916bf7`, resident `tts-server`, 1.7 B talker**, with `TTS_BACKEND=llama-tts-cli` kept as the rollback. This branch now carries that decision (new *Decision: arm A for live output* section), the corrected addressing law (the pair may not meet at a host-loopback publish — `host.containers.internal` is the gateway, so a `127.0.0.1` bind is unreachable from a bridge container), and the two follow-ups left deliberately open (0.6 B talker, ICL vs embedding-only). Build worktree for this branch: `/home/blank/.omp/wt/wt-tts-trial-docs`. --- cwd: /home/blank/.omp/wt/wt-20261017-181502-6d1fc1f · session: 01a14c93-8870-75d9-9900-9a731559feef
dank added 1 commit 2026-10-18 16:20:39 -07:00
Both candidate :8880 engines were measured on Dankclaw's CPU (no GPU,
no container, nothing deployed). arm A qwentts.cpp 5916bf7 resident at
RTF 0.55-0.57 warm / 0.67 ICL / 1.65 cold; arm B mainline llama-tts at
0.93-0.99 with a +-0.25 spread. Records the four configuration laws the
trial produced (boot-warm every voice, .spk+.rvq replay is byte-identical,
streaming != buffered, /v1/audio/voices rejects a lone spk_b64), the
likeness measurement's resolution limit, and re-keys the stale
8890/voiceshim shape on that page to frank-tts on 7772. The engine pick
stays open: the likeness metric cannot rank clones, so the audition clips
decide.
dank added 3 commits 2026-10-18 16:54:15 -07:00
Frank heard every audition pair and passed it, so likeness stopped being the
tiebreak and the engine was picked on the live-output criterion: arm A
qwentts.cpp 5916bf7, resident tts-server, 1.7 B talker, llama-tts-cli kept as
the rollback. Records the discriminators (first audio byte, warm RTF, spread),
the three consequences that are configuration (boot-warm every voice, no cpuset
pin, WAV-only registration), the two follow-ups left open on purpose, and the
corrected addressing law: a host-loopback publish is unreachable from a bridge
container, so the pair shares one quadlet network.
# Conflicts:
#	serverAppConfigs-docs/log.md
dank merged commit e13f875c94 into main 2026-10-18 17:15:00 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#10