docs(frank-tts): what makes a locally built voice image stale, and what the warm text must survive

Three pages, all from faults found this session:

- `LLM/deployment.md` gains "Locally built images": which images have no
  registry digest, what the `cfhash` skip is for, why it must cover every COPY'd
  file (the silent shape: build skipped, /health green, old code serving), and
  the base-image half of the law — the engine's `useradd`, because the pinned
  ubuntu:24.04 has no `adduser`.
- `Dankclaw/Frank TTS Engine.md` troubleshooting gains the two lines an operator
  actually needs: every voice cold on a fresh boot with a hand-run hook
  succeeding (= `WARM_TEXT` arrived quoted, `podman exec … env | grep WARM_TEXT`
  proves it), and a 502 from the façade beside a 200 `/health` from the engine
  (model loaded ≠ voices registered). The stale-warmer row is re-keyed to the
  new rebuild rule.
- `Dankclaw/Voice Packs.md` step 4 was wrong: it told the next agent to restart
  the engine *instead of* the warm hook because the hook "only registers packs it
  has not seen". It registers every pack on every run and the engine replaces a
  name it already holds — `tools/tts-server.cpp:148` at the pinned commit:
  "Re-registering a name replaces the previous entry." A restart is what an
  engine bump needs (the registry is memory-only), never worse, only slower.
- `Dankclaw/Frank TTS.md` deploy command re-keyed to `./scripts/deploy.sh
  dankclaw --service frank-tts`, the way it is actually deployed; the raw
  `deploy-remote.sh` form is plumbing.
This commit is contained in:
Frank committed 2026-10-18 18:07:19 -07:00
1 parent 59e021da2c
commit 7178e1c789
4 files changed
+44 -8

No files matched your search

@@ -109,9 +109,9 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline. Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline.
1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls). 1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls).
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw frank-tts-engine`. Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms. 2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it. 3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it.
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/voices` is the authoritative list. 4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact). 5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact).
6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not. 6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not.
7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]]. 7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]].
@@ -126,4 +126,6 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock
| Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) | | Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
| `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work | | `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
| A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design | | A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | rebuild: `build_compose_image` skips on an unchanged `Containerfile.engine`, and the `COPY` of the warmer changed it | | `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |
+1 -1
View File
@@ -115,7 +115,7 @@ shows up as a failed synthesis, never as a down probe.
## Deploying it ## Deploying it
`scripts/deploy-remote.sh dankclaw frank-tts` (or `frank-tts-engine frank-tts`): `./scripts/deploy.sh dankclaw --service frank-tts` (or `--service frank-tts-engine --service frank-tts` for the pair):
builds the façade image, installs **both** Quadlet units — including the builds the façade image, installs **both** Quadlet units — including the
engine's `.network` file, because a referenced `Network=` file the generator engine's `.network` file, because a referenced `Network=` file the generator
cannot find means the façade unit is not produced at all — creates cannot find means the façade unit is not produced at all — creates
+10 -4
View File
@@ -197,10 +197,16 @@ loud, not a bug. Work through it in order:
without moving the commit is the mistake to avoid. without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old 4. Rebuild all three packs with `--force` (the retired directories keep the old
bytes readable beside their replacements until a human clears them), then bytes readable beside their replacements until a human clears them), then
restart the engine rather than using the warm hook: the hook only registers re-run the warm hook: `podman exec frank-tts-engine /app/warm-voices.sh`. That
packs it has **not** seen, and every voice here is already registered against hook registers **every** pack it finds on every run, and the engine replaces a
the old artifacts. `systemctl --user restart frank-tts-engine.service` name it already holds — `tools/tts-server.cpp:148` at the pinned commit says so
re-warms them all from the new bytes. in its own words: *"Re-registering a name replaces the previous entry."* So the
rebuilt `reference.wav` is what the running engine speaks afterwards, and the
hook's cold/warm pass is what re-snapshots it. A restart
(`systemctl --user restart frank-tts-engine.service`) is what an ENGINE bump
needs — the registry is memory-only and lives in that process — and the unit
runs the same hook on every start anyway, so a restart is never worse, only
slower.
5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing 5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing
law, so it needs the same arithmetic update as the env file. The seam's law, so it needs the same arithmetic update as the env file. The seam's
engine-bump test is exactly this scenario with a wrong `.spk` size, and its engine-bump test is exactly this scenario with a wrong `.spk` size, and its
+28
View File
@@ -116,6 +116,34 @@ On a later ordinary deployment, a service remains pending only while its
recorded revision differs from the checkout and the planner still finds a recorded revision differs from the checkout and the planner still finds a
change in that service's inputs. change in that service's inputs.
## Locally built images
`build_compose_image` in `scripts/deploy-remote.sh` builds the images that have
no registry digest — `comfyui-rocm`, `franksdank/loki`, `franksdank/alloy` and
the voice pair `franksdank/frank-tts` / `franksdank/frank-tts-engine` — and
**skips** the build when a `cfhash` label on the installed image matches a hash
it computes from the checkout now. That skip is what keeps the engine's
ten-minute `git clone --recursive` off every re-deploy, so the rule has to be
honest about what the image bakes:
- The hash covers the `Containerfile` **plus the files its `COPY` lines pull
in**, named as `extra=(…)` in each case arm. Hashing the `Containerfile`
alone was wrong in practice: editing `dankclaw/voiceservice/app.py` or
`tts-engine-warm.sh` left the label matching, `podman build` never ran, and
the redeployed container still carried the previous code while `/health`
answered — invisible to every gate. Corrected 2026-10-18;
`tests/frank_tts_engine.bats` drives the function against a stub `podman`
(first deploy builds, unchanged tree does not, a `COPY`'d file rebuilds).
- A base-image assumption is part of the image's identity too. The engine's
runtime stage creates its unprivileged user with `useradd`, not `adduser`,
because the pinned `ubuntu:24.04` base ships no `adduser` — checked by
running the command against the digest on 2026-10-18. That failure mode is a
build that dies at the `RUN` line, so it is loud, unlike the stale-hash one.
Both engines keep serving during a rebuild: `build_compose_image` runs after
`install_quadlet` only for the LLM-style handlers, and the voice deploy gates on
recorded health rather than a fixed sleep — see [[Frank TTS Engine]].
## Manual recovery ## Manual recovery
If the LLM handler fails after the target pull: If the LLM handler fails after the target pull: