From 7178e1c7899a247cc31cda2a22759cb49a4df371 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 18 Oct 2026 18:07:19 -0700 Subject: [PATCH] docs(frank-tts): what makes a locally built voice image stale, and what the warm text must survive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pages, all from faults found this session: - `LLM/deployment.md` gains "Locally built images": which images have no registry digest, what the `cfhash` skip is for, why it must cover every COPY'd file (the silent shape: build skipped, /health green, old code serving), and the base-image half of the law — the engine's `useradd`, because the pinned ubuntu:24.04 has no `adduser`. - `Dankclaw/Frank TTS Engine.md` troubleshooting gains the two lines an operator actually needs: every voice cold on a fresh boot with a hand-run hook succeeding (= `WARM_TEXT` arrived quoted, `podman exec … env | grep WARM_TEXT` proves it), and a 502 from the façade beside a 200 `/health` from the engine (model loaded ≠ voices registered). The stale-warmer row is re-keyed to the new rebuild rule. - `Dankclaw/Voice Packs.md` step 4 was wrong: it told the next agent to restart the engine *instead of* the warm hook because the hook "only registers packs it has not seen". It registers every pack on every run and the engine replaces a name it already holds — `tools/tts-server.cpp:148` at the pinned commit: "Re-registering a name replaces the previous entry." A restart is what an engine bump needs (the registry is memory-only), never worse, only slower. - `Dankclaw/Frank TTS.md` deploy command re-keyed to `./scripts/deploy.sh dankclaw --service frank-tts`, the way it is actually deployed; the raw `deploy-remote.sh` form is plumbing. --- .../Dankclaw/Frank TTS Engine.md | 8 ++++-- serverAppConfigs-docs/Dankclaw/Frank TTS.md | 2 +- serverAppConfigs-docs/Dankclaw/Voice Packs.md | 14 +++++++--- serverAppConfigs-docs/LLM/deployment.md | 28 +++++++++++++++++++ 4 files changed, 44 insertions(+), 8 deletions(-) diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md index e23aeac..209c12c 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md @@ -109,9 +109,9 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline. 1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls). -2. **Deploy the engine** — `./scripts/deploy.sh dankclaw frank-tts-engine`. Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms. +2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes ` [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms. 3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it. -4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/voices` is the authoritative list. +4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`). 5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact). 6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not. 7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]]. @@ -126,4 +126,6 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock | Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) | | `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work | | A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design | -| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | rebuild: `build_compose_image` skips on an unchanged `Containerfile.engine`, and the `COPY` of the warmer changed it | +| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] | +| The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this | +| A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces | diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index 62322ef..578e53e 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -115,7 +115,7 @@ shows up as a failed synthesis, never as a down probe. ## Deploying it -`scripts/deploy-remote.sh dankclaw frank-tts` (or `frank-tts-engine frank-tts`): +`./scripts/deploy.sh dankclaw --service frank-tts` (or `--service frank-tts-engine --service frank-tts` for the pair): builds the façade image, installs **both** Quadlet units — including the engine's `.network` file, because a referenced `Network=` file the generator cannot find means the façade unit is not produced at all — creates diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md index ce23c2d..979b935 100644 --- a/serverAppConfigs-docs/Dankclaw/Voice Packs.md +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -197,10 +197,16 @@ loud, not a bug. Work through it in order: without moving the commit is the mistake to avoid. 4. Rebuild all three packs with `--force` (the retired directories keep the old bytes readable beside their replacements until a human clears them), then - restart the engine rather than using the warm hook: the hook only registers - packs it has **not** seen, and every voice here is already registered against - the old artifacts. `systemctl --user restart frank-tts-engine.service` - re-warms them all from the new bytes. + re-run the warm hook: `podman exec frank-tts-engine /app/warm-voices.sh`. That + hook registers **every** pack it finds on every run, and the engine replaces a + name it already holds — `tools/tts-server.cpp:148` at the pinned commit says so + in its own words: *"Re-registering a name replaces the previous entry."* So the + rebuilt `reference.wav` is what the running engine speaks afterwards, and the + hook's cold/warm pass is what re-snapshots it. A restart + (`systemctl --user restart frank-tts-engine.service`) is what an ENGINE bump + needs — the registry is memory-only and lives in that process — and the unit + runs the same hook on every start anyway, so a restart is never worse, only + slower. 5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing law, so it needs the same arithmetic update as the env file. The seam's engine-bump test is exactly this scenario with a wrong `.spk` size, and its diff --git a/serverAppConfigs-docs/LLM/deployment.md b/serverAppConfigs-docs/LLM/deployment.md index 92be135..93f6d07 100644 --- a/serverAppConfigs-docs/LLM/deployment.md +++ b/serverAppConfigs-docs/LLM/deployment.md @@ -116,6 +116,34 @@ On a later ordinary deployment, a service remains pending only while its recorded revision differs from the checkout and the planner still finds a change in that service's inputs. +## Locally built images + +`build_compose_image` in `scripts/deploy-remote.sh` builds the images that have +no registry digest — `comfyui-rocm`, `franksdank/loki`, `franksdank/alloy` and +the voice pair `franksdank/frank-tts` / `franksdank/frank-tts-engine` — and +**skips** the build when a `cfhash` label on the installed image matches a hash +it computes from the checkout now. That skip is what keeps the engine's +ten-minute `git clone --recursive` off every re-deploy, so the rule has to be +honest about what the image bakes: + +- The hash covers the `Containerfile` **plus the files its `COPY` lines pull + in**, named as `extra=(…)` in each case arm. Hashing the `Containerfile` + alone was wrong in practice: editing `dankclaw/voiceservice/app.py` or + `tts-engine-warm.sh` left the label matching, `podman build` never ran, and + the redeployed container still carried the previous code while `/health` + answered — invisible to every gate. Corrected 2026-10-18; + `tests/frank_tts_engine.bats` drives the function against a stub `podman` + (first deploy builds, unchanged tree does not, a `COPY`'d file rebuilds). +- A base-image assumption is part of the image's identity too. The engine's + runtime stage creates its unprivileged user with `useradd`, not `adduser`, + because the pinned `ubuntu:24.04` base ships no `adduser` — checked by + running the command against the digest on 2026-10-18. That failure mode is a + build that dies at the `RUN` line, so it is loud, unlike the stale-hash one. + +Both engines keep serving during a rebuild: `build_compose_image` runs after +`install_quadlet` only for the LLM-style handlers, and the voice deploy gates on +recorded health rather than a fixed sleep — see [[Frank TTS Engine]]. + ## Manual recovery If the LLM handler fails after the target pull: