docs(frank-tts): the cold-voice ladder did not survive the deployment
The first real boot gave the chance to measure what warming actually buys, and the trial's numbers were wrong: - Empty registry, register only, then time: the FIRST synthesis of a freshly registered voice cost 1 883 ms; the next two 1 283 and 1 226 ms. A ~110-word request cost 9.4 s and then 9.8 s. - So the published 6.2–8.0 s cold / first-audio-byte 5 530 ms / later-requests 0.5 ms ladder does not reproduce, and request time tracks the audio generated rather than snapshot state. The 0.5 ms figure is an internal state-restore cost, never an end-to-end request time. - Warming still earns its place for a different reason than the page claimed: the registry is memory-only and re-registering does not warm, so a boot without the hook leaves the façade answering 400 for every name. The saving is ~0.6 s. Law 1 of [[Frank TTS Engine]] is re-titled the registry law and carries the measured numbers; its example journal line is a real one from today; the troubleshooting row for "first request of the day is slow" — the retired law's symptom — is replaced by the argument-cap fault that actually happened. The same ladder in the research brief is annotated rather than erased, and [[Voice Packs]] step 6 stops claiming the hook registers only packs it has not seen (the engine replaces a name it already holds).
This commit is contained in:
@@ -50,17 +50,17 @@ The spread is the whole argument: at ±0.01 the engine finishes synthesizing 1.5
|
||||
|
||||
## Three laws this service lives under
|
||||
|
||||
**1. The cold-voice law → the unit warms at boot.** The **first** speech request for a registered voice decodes the whole reference clip to snapshot its codec state: **6.2–8.0 s, first audio byte at 5 530 ms**. Every later request for that voice restores the snapshot in **0.5 ms**. Registrations live in memory only — a restart forgets every voice — and re-registering does **not** warm a voice, only synthesizing does. So `frank-tts-engine.container` carries
|
||||
**1. The registry law → the unit warms at boot.** The voice registry is **memory only**: a restart forgets every voice, and re-registering one does **not** warm it — only synthesizing does. A boot that registers nothing leaves the façade answering 400 for every name, which is why `frank-tts-engine.container` carries
|
||||
|
||||
```
|
||||
ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh
|
||||
TimeoutStartSec=900
|
||||
```
|
||||
|
||||
and the warm pass runs on every start including boot. Without it, whoever asks for audio first that day pays 5.5 s of silence. The `-` prefix is load-bearing: with `Restart=always`, a warmer that exits non-zero would restart the container and crash-loop the endpoint forever. The hook is `podman exec` rather than a host script **because this container publishes nothing** — there is no address the host could dial — and a consequence worth knowing: an exec inherits the unit's `EnvironmentFile`, so the warmer's knobs (`VOICES_DIR`, `HEALTH_WAIT`, `WARM_TEXT`, `REQUEST_TIMEOUT`) live in `scripts/frank-tts-engine.env` beside the engine's own. The deployer runs the same hook a second time so a failed warm becomes a deploy result rather than only a journal line, and so the per-voice numbers land on the console:
|
||||
and the warm pass runs on every start including boot. What it buys is registration (mandatory) plus a measurably smaller first request — and the size of that saving is **not** the number the trial published. **Re-measured on the deployed build, 2026-10-18,** against an empty registry (register, then time the synthesis): the first request for `narrator1` cost **1 883 ms**, the second **1 283 ms**, the third **1 226 ms** — about **0.6 s** of saved silence. A ~110-word request cost **9.4 s** and then **9.8 s**, so request time tracks the audio generated, not snapshot state. The trial's **6.2–8.0 s / first audio byte 5 530 ms / later requests 0.5 ms** ladder does **not** reproduce here; the 0.5 ms figure is an internal state-restore cost that was never an end-to-end request time, and the honest claim is the ~0.6 s above. The `-` prefix is load-bearing: with `Restart=always`, a warmer that exits non-zero would restart the container and crash-loop the endpoint forever. The hook is `podman exec` rather than a host script **because this container publishes nothing** — there is no address the host could dial — and a consequence worth knowing: an exec inherits the unit's `EnvironmentFile`, so the warmer's knobs (`VOICES_DIR`, `HEALTH_WAIT`, `WARM_TEXT`, `REQUEST_TIMEOUT`) live in `scripts/frank-tts-engine.env` beside the engine's own. The deployer runs the same hook a second time so a failed warm becomes a deploy result rather than only a journal line, and so the per-voice numbers land on the console:
|
||||
|
||||
```
|
||||
frank-tts-engine-warm: voice narrator1 cold=6821ms warm=412ms
|
||||
frank-tts-engine-warm: voice narrator1 cold=1402ms warm=1812ms
|
||||
frank-tts-engine-warm: 3 voice(s) registered and warm
|
||||
```
|
||||
|
||||
@@ -122,7 +122,7 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock
|
||||
|---|---|---|
|
||||
| Unit restarts every 10 s, journal says `cannot read /models/…` | weights missing or the mount is empty | run the fetcher; re-deploy |
|
||||
| Unit `active`, but every voice name returns 400 from the façade | the engine restarted and lost its in-memory registry, or the warm pass failed | `journalctl --user -u frank-tts-engine` for the warmer's lines; `podman exec frank-tts-engine /app/warm-voices.sh` |
|
||||
| First request of the day is slow, the rest are fast | the warm pass did not run (the hook carries `-`, so it cannot fail the unit) | look for `did not answer` in the journal — the warmer gave up while the model was still loading |
|
||||
| Every voice logs `registration failed (HTTP no response)`, with `/usr/bin/curl: Argument list too long` on the line above it | the hook handed the registration body to curl as one argument: the kernel caps a single argument at 131 072 bytes (`MAX_ARG_STRLEN`), and a shipped `reference.wav` base64's to about 1.27 MB | the body goes through a temp file and `--data-binary @file` (2026-10-18). Any pack over ~96 KB trips this, so no small fixture can see it — the seam ships a 300 KB pack for exactly that reason |
|
||||
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
|
||||
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
|
||||
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
|
||||
|
||||
@@ -169,10 +169,14 @@ invariant to every crop in the tracked pack list.
|
||||
curl -s localhost:7772/v1/audio/voices # the façade's list, not the engine's
|
||||
```
|
||||
|
||||
The hook walks `/voices`, registers any pack with a `reference.wav` it has not
|
||||
seen, and logs `voice <name> cold=…ms warm=…ms` to the unit's journal. The
|
||||
rootless unit restart — `systemctl --user restart frank-tts-engine.service` —
|
||||
is for when the unit or its env changed, not for a new pack.
|
||||
The hook walks `/voices` and registers **every** pack that has a
|
||||
`reference.wav`, on every run — the engine replaces a name it already holds
|
||||
(`tools/tts-server.cpp:148` at the pinned commit: *"Re-registering a name
|
||||
replaces the previous entry."*) — then synthesizes twice per voice and logs
|
||||
`voice <name> cold=…ms warm=…ms` to the unit's journal. The two numbers should
|
||||
be close; see law 1 of [[Frank TTS Engine]] for what they measure. The rootless
|
||||
unit restart — `systemctl --user restart frank-tts-engine.service` — is for when
|
||||
the unit or its env changed, not for a new pack.
|
||||
|
||||
## When the extractor's output size changes on an engine bump
|
||||
|
||||
|
||||
@@ -258,11 +258,17 @@ HTTP path at all — that is what the façade's CLI backend exists for.
|
||||
|
||||
Four laws came out of the trial, each worth a line of configuration:
|
||||
|
||||
- **Boot-warm every voice.** The first request for a registered voice decodes
|
||||
the whole reference to snapshot codec state: 6.2–8.0 s, first audio byte at
|
||||
5,530 ms. Later requests restore that state in 0.5 ms. An engine unit must
|
||||
run one throwaway synthesis per voice at start, or the first request of the
|
||||
day pays five seconds.
|
||||
- **Boot-warm every voice.** The trial measured the first request for a
|
||||
registered voice at 6.2–8.0 s (first audio byte 5,530 ms) and later requests
|
||||
restoring the snapshot in 0.5 ms. **That ladder did not survive deployment:**
|
||||
measured 2026-10-18 on the built container against an empty registry — register,
|
||||
then time — the first synthesis of a freshly registered voice cost **1,883 ms**
|
||||
against **1,226–1,283 ms** for the requests after it, and a ~110-word request
|
||||
cost **9.4 s** then **9.8 s**, so request time tracks the audio generated rather
|
||||
than snapshot state. What still holds is the reason to warm at all: the registry
|
||||
is memory-only and re-registering does not warm, so an engine unit must run one
|
||||
throwaway synthesis per voice at start. It buys registration and about 0.6 s,
|
||||
not five seconds.
|
||||
- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference
|
||||
WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T,
|
||||
11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a
|
||||
|
||||
@@ -2643,3 +2643,42 @@ every restart since the cutover.
|
||||
and a new unit on a host that serves people — Frank's call), no voice pack
|
||||
built on the host, no Playwright run (this branch publishes no host port and
|
||||
touches no vhost), and Gitea **#2** phase 1 stays open until 7772 answers.
|
||||
|
||||
## 2026-10-18 — frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap
|
||||
|
||||
- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`. The
|
||||
payload took 18 s to fetch for the 1.4 GiB pair (byte count + Hub LFS sha256,
|
||||
`--verify-only` OK), and the image built clean — the first real proof of the
|
||||
`useradd` fix: `STEP 3/13: RUN useradd --system --uid 10000 …` passed with only
|
||||
the documented cosmetic `SYS_UID_MAX` warning, and the unit was `healthy` after
|
||||
31 s.
|
||||
- Then the deploy reported `0 voice(s) warm`:
|
||||
`/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long`. The hook
|
||||
handed the registration body to curl as ONE argument; the kernel caps a single
|
||||
argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped 955 KB
|
||||
`reference.wav` base64's to about 1.27 MB, so curl never executed and the hook
|
||||
logged that as "HTTP no response". Fourth silent fault of the voice stack, same
|
||||
species as the other three: `/health` stayed green throughout.
|
||||
- Fix: the body goes to a `mktemp` file and curl gets `@file`. The regression test
|
||||
is the honest shape — the seam's stub engine is a real listener, `curl` is the
|
||||
real binary, the fixture pack is 300 KB — so it fails with the same kernel error
|
||||
the host produced (proved red first, green after). Every shipped pack sits well
|
||||
over the ~96 KB cliff, so no small fixture could ever have seen this.
|
||||
- Live proof on the deployed container with the three real packs:
|
||||
`narrator1 cold=1175ms warm=1874ms`, `narrator2 1373/1892`, `narrator3
|
||||
1816/1487`, exit 0, 15.2 s for the whole pass.
|
||||
- **A trial law did not survive deployment.** Against an empty registry —
|
||||
register only, then time — the FIRST synthesis of a freshly registered voice
|
||||
cost **1 883 ms**, the next two **1 283** and **1 226 ms**; a ~110-word request
|
||||
cost **9.4 s** then **9.8 s**. The trial's 6.2–8.0 s cold / 0.5 ms restore
|
||||
ladder, which the warmer's own header repeated, is retired: warming buys
|
||||
registration — mandatory, because the registry is memory-only — plus about
|
||||
**0.6 s**. Corrected in [[Frank TTS Engine]] law 1 and its troubleshooting
|
||||
table, in [[Local TTS and Voice Cloning]], and in [[Voice Packs]].
|
||||
- Host state, honestly: the engine is `active` with three voices warm, but the hook
|
||||
fix is not in the image yet, so the next restart re-breaks it until this branch
|
||||
is merged and re-deployed. `frank-tts` (the 7772 façade) is still undeployed.
|
||||
- Deliberately NOT done in this window: the `frank-tts` deploy, any Prometheus
|
||||
probe for the voice pair (neither container exposes `/metrics`; the blackbox
|
||||
question is still open), and the podcast render — open-notebook is on `main` but
|
||||
not installed.
|
||||
|
||||
Reference in New Issue
Block a user