docs(frank-tts): what makes a locally built voice image stale, and what the warm text must survive

Three pages, all from faults found this session:

- `LLM/deployment.md` gains "Locally built images": which images have no
  registry digest, what the `cfhash` skip is for, why it must cover every COPY'd
  file (the silent shape: build skipped, /health green, old code serving), and
  the base-image half of the law — the engine's `useradd`, because the pinned
  ubuntu:24.04 has no `adduser`.
- `Dankclaw/Frank TTS Engine.md` troubleshooting gains the two lines an operator
  actually needs: every voice cold on a fresh boot with a hand-run hook
  succeeding (= `WARM_TEXT` arrived quoted, `podman exec … env | grep WARM_TEXT`
  proves it), and a 502 from the façade beside a 200 `/health` from the engine
  (model loaded ≠ voices registered). The stale-warmer row is re-keyed to the
  new rebuild rule.
- `Dankclaw/Voice Packs.md` step 4 was wrong: it told the next agent to restart
  the engine *instead of* the warm hook because the hook "only registers packs it
  has not seen". It registers every pack on every run and the engine replaces a
  name it already holds — `tools/tts-server.cpp:148` at the pinned commit:
  "Re-registering a name replaces the previous entry." A restart is what an
  engine bump needs (the registry is memory-only), never worse, only slower.
- `Dankclaw/Frank TTS.md` deploy command re-keyed to `./scripts/deploy.sh
  dankclaw --service frank-tts`, the way it is actually deployed; the raw
  `deploy-remote.sh` form is plumbing.
This commit is contained in:
Frank
2026-10-18 18:07:19 -07:00
parent 59e021da2c
commit 7178e1c789
4 changed files with 44 additions and 8 deletions
@@ -109,9 +109,9 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline.
1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls).
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw frank-tts-engine`. Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it.
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/voices` is the authoritative list.
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact).
6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not.
7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]].
@@ -126,4 +126,6 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | rebuild: `build_compose_image` skips on an unchanged `Containerfile.engine`, and the `COPY` of the warmer changed it |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |
+1 -1
View File
@@ -115,7 +115,7 @@ shows up as a failed synthesis, never as a down probe.
## Deploying it
`scripts/deploy-remote.sh dankclaw frank-tts` (or `frank-tts-engine frank-tts`):
`./scripts/deploy.sh dankclaw --service frank-tts` (or `--service frank-tts-engine --service frank-tts` for the pair):
builds the façade image, installs **both** Quadlet units — including the
engine's `.network` file, because a referenced `Network=` file the generator
cannot find means the façade unit is not produced at all — creates
+10 -4
View File
@@ -197,10 +197,16 @@ loud, not a bug. Work through it in order:
without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old
bytes readable beside their replacements until a human clears them), then
restart the engine rather than using the warm hook: the hook only registers
packs it has **not** seen, and every voice here is already registered against
the old artifacts. `systemctl --user restart frank-tts-engine.service`
re-warms them all from the new bytes.
re-run the warm hook: `podman exec frank-tts-engine /app/warm-voices.sh`. That
hook registers **every** pack it finds on every run, and the engine replaces a
name it already holds — `tools/tts-server.cpp:148` at the pinned commit says so
in its own words: *"Re-registering a name replaces the previous entry."* So the
rebuilt `reference.wav` is what the running engine speaks afterwards, and the
hook's cold/warm pass is what re-snapshots it. A restart
(`systemctl --user restart frank-tts-engine.service`) is what an ENGINE bump
needs — the registry is memory-only and lives in that process — and the unit
runs the same hook on every start anyway, so a restart is never worse, only
slower.
5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing
law, so it needs the same arithmetic update as the env file. The seam's
engine-bump test is exactly this scenario with a wrong `.spk` size, and its
+28
View File
@@ -116,6 +116,34 @@ On a later ordinary deployment, a service remains pending only while its
recorded revision differs from the checkout and the planner still finds a
change in that service's inputs.
## Locally built images
`build_compose_image` in `scripts/deploy-remote.sh` builds the images that have
no registry digest — `comfyui-rocm`, `franksdank/loki`, `franksdank/alloy` and
the voice pair `franksdank/frank-tts` / `franksdank/frank-tts-engine` — and
**skips** the build when a `cfhash` label on the installed image matches a hash
it computes from the checkout now. That skip is what keeps the engine's
ten-minute `git clone --recursive` off every re-deploy, so the rule has to be
honest about what the image bakes:
- The hash covers the `Containerfile` **plus the files its `COPY` lines pull
in**, named as `extra=(…)` in each case arm. Hashing the `Containerfile`
alone was wrong in practice: editing `dankclaw/voiceservice/app.py` or
`tts-engine-warm.sh` left the label matching, `podman build` never ran, and
the redeployed container still carried the previous code while `/health`
answered — invisible to every gate. Corrected 2026-10-18;
`tests/frank_tts_engine.bats` drives the function against a stub `podman`
(first deploy builds, unchanged tree does not, a `COPY`'d file rebuilds).
- A base-image assumption is part of the image's identity too. The engine's
runtime stage creates its unprivileged user with `useradd`, not `adduser`,
because the pinned `ubuntu:24.04` base ships no `adduser` — checked by
running the command against the digest on 2026-10-18. That failure mode is a
build that dies at the `RUN` line, so it is loud, unlike the stale-hash one.
Both engines keep serving during a rebuild: `build_compose_image` runs after
`install_quadlet` only for the LLM-style handlers, and the voice deploy gates on
recorded health rather than a fixed sleep — see [[Frank TTS Engine]].
## Manual recovery
If the LLM handler fails after the target pull: