docs(frank-tts): what makes a locally built voice image stale, and what the warm text must survive
Three pages, all from faults found this session: - `LLM/deployment.md` gains "Locally built images": which images have no registry digest, what the `cfhash` skip is for, why it must cover every COPY'd file (the silent shape: build skipped, /health green, old code serving), and the base-image half of the law — the engine's `useradd`, because the pinned ubuntu:24.04 has no `adduser`. - `Dankclaw/Frank TTS Engine.md` troubleshooting gains the two lines an operator actually needs: every voice cold on a fresh boot with a hand-run hook succeeding (= `WARM_TEXT` arrived quoted, `podman exec … env | grep WARM_TEXT` proves it), and a 502 from the façade beside a 200 `/health` from the engine (model loaded ≠ voices registered). The stale-warmer row is re-keyed to the new rebuild rule. - `Dankclaw/Voice Packs.md` step 4 was wrong: it told the next agent to restart the engine *instead of* the warm hook because the hook "only registers packs it has not seen". It registers every pack on every run and the engine replaces a name it already holds — `tools/tts-server.cpp:148` at the pinned commit: "Re-registering a name replaces the previous entry." A restart is what an engine bump needs (the registry is memory-only), never worse, only slower. - `Dankclaw/Frank TTS.md` deploy command re-keyed to `./scripts/deploy.sh dankclaw --service frank-tts`, the way it is actually deployed; the raw `deploy-remote.sh` form is plumbing.
This commit is contained in:
@@ -109,9 +109,9 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
|
||||
Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline.
|
||||
|
||||
1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls).
|
||||
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw frank-tts-engine`. Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
|
||||
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
|
||||
3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it.
|
||||
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/voices` is the authoritative list.
|
||||
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
|
||||
5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact).
|
||||
6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not.
|
||||
7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]].
|
||||
@@ -126,4 +126,6 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock
|
||||
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
|
||||
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
|
||||
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
|
||||
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | rebuild: `build_compose_image` skips on an unchanged `Containerfile.engine`, and the `COPY` of the warmer changed it |
|
||||
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
|
||||
| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
|
||||
| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |
|
||||
|
||||
@@ -115,7 +115,7 @@ shows up as a failed synthesis, never as a down probe.
|
||||
|
||||
## Deploying it
|
||||
|
||||
`scripts/deploy-remote.sh dankclaw frank-tts` (or `frank-tts-engine frank-tts`):
|
||||
`./scripts/deploy.sh dankclaw --service frank-tts` (or `--service frank-tts-engine --service frank-tts` for the pair):
|
||||
builds the façade image, installs **both** Quadlet units — including the
|
||||
engine's `.network` file, because a referenced `Network=` file the generator
|
||||
cannot find means the façade unit is not produced at all — creates
|
||||
|
||||
@@ -197,10 +197,16 @@ loud, not a bug. Work through it in order:
|
||||
without moving the commit is the mistake to avoid.
|
||||
4. Rebuild all three packs with `--force` (the retired directories keep the old
|
||||
bytes readable beside their replacements until a human clears them), then
|
||||
restart the engine rather than using the warm hook: the hook only registers
|
||||
packs it has **not** seen, and every voice here is already registered against
|
||||
the old artifacts. `systemctl --user restart frank-tts-engine.service`
|
||||
re-warms them all from the new bytes.
|
||||
re-run the warm hook: `podman exec frank-tts-engine /app/warm-voices.sh`. That
|
||||
hook registers **every** pack it finds on every run, and the engine replaces a
|
||||
name it already holds — `tools/tts-server.cpp:148` at the pinned commit says so
|
||||
in its own words: *"Re-registering a name replaces the previous entry."* So the
|
||||
rebuilt `reference.wav` is what the running engine speaks afterwards, and the
|
||||
hook's cold/warm pass is what re-snapshots it. A restart
|
||||
(`systemctl --user restart frank-tts-engine.service`) is what an ENGINE bump
|
||||
needs — the registry is memory-only and lives in that process — and the unit
|
||||
runs the same hook on every start anyway, so a restart is never worse, only
|
||||
slower.
|
||||
5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing
|
||||
law, so it needs the same arithmetic update as the env file. The seam's
|
||||
engine-bump test is exactly this scenario with a wrong `.spk` size, and its
|
||||
|
||||
@@ -116,6 +116,34 @@ On a later ordinary deployment, a service remains pending only while its
|
||||
recorded revision differs from the checkout and the planner still finds a
|
||||
change in that service's inputs.
|
||||
|
||||
## Locally built images
|
||||
|
||||
`build_compose_image` in `scripts/deploy-remote.sh` builds the images that have
|
||||
no registry digest — `comfyui-rocm`, `franksdank/loki`, `franksdank/alloy` and
|
||||
the voice pair `franksdank/frank-tts` / `franksdank/frank-tts-engine` — and
|
||||
**skips** the build when a `cfhash` label on the installed image matches a hash
|
||||
it computes from the checkout now. That skip is what keeps the engine's
|
||||
ten-minute `git clone --recursive` off every re-deploy, so the rule has to be
|
||||
honest about what the image bakes:
|
||||
|
||||
- The hash covers the `Containerfile` **plus the files its `COPY` lines pull
|
||||
in**, named as `extra=(…)` in each case arm. Hashing the `Containerfile`
|
||||
alone was wrong in practice: editing `dankclaw/voiceservice/app.py` or
|
||||
`tts-engine-warm.sh` left the label matching, `podman build` never ran, and
|
||||
the redeployed container still carried the previous code while `/health`
|
||||
answered — invisible to every gate. Corrected 2026-10-18;
|
||||
`tests/frank_tts_engine.bats` drives the function against a stub `podman`
|
||||
(first deploy builds, unchanged tree does not, a `COPY`'d file rebuilds).
|
||||
- A base-image assumption is part of the image's identity too. The engine's
|
||||
runtime stage creates its unprivileged user with `useradd`, not `adduser`,
|
||||
because the pinned `ubuntu:24.04` base ships no `adduser` — checked by
|
||||
running the command against the digest on 2026-10-18. That failure mode is a
|
||||
build that dies at the `RUN` line, so it is loud, unlike the stale-hash one.
|
||||
|
||||
Both engines keep serving during a rebuild: `build_compose_image` runs after
|
||||
`install_quadlet` only for the LLM-style handlers, and the voice deploy gates on
|
||||
recorded health rather than a fixed sleep — see [[Frank TTS Engine]].
|
||||
|
||||
## Manual recovery
|
||||
|
||||
If the LLM handler fails after the target pull:
|
||||
|
||||
Reference in New Issue
Block a user