fix(frank-tts): the warm hook died on the kernel argument cap; the cold-voice law is retired by measurement #15

Merged
dank merged 2 commits from fix/frank-tts-warm-argv into main 2026-10-18 18:50:01 -07:00
Owner

The fault

The first deploy of frank-tts-engine built the image (the useradd fix holds — STEP 3/13 passed, unit healthy after 31 s), and then reported:

/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long
frank-tts-engine-warm: voice 'narrator1': registration failed (HTTP no response) — the shim will answer 400 for this name
… narrator2, narrator3 …
frank-tts-engine-warm: 0 voice(s) warm, at least one failed (see the lines above)

with /health answering 200 the whole time. The hook built {"name":…,"wav_b64":…} in a shell variable and handed it to curl as one argument. The kernel caps a single argument at MAX_ARG_STRLEN = 131,072 bytes; a shipped reference.wav (955 KB) base64's to ~1.27 MB, so curl was never executed and the hook reported the absence of a response as an HTTP failure.

Fourth silent fault in the voice stack, same species as the three in #14: the failure looked like health.

Fix

Registration body → mktemp file, curl reads @file; the audio enters through a redirection instead of a command substitution, so the script never holds a 1.3 MB string at all. One file for the pass, reused per voice, removed on exit (container /tmp, not host state).

Test

The seam's stub engine is a real listener and curl is the real binary, so the fixture pack (300 KB) trips the actual kernel limit: the test fails with the host's own error before the fix, passes after. Proved red → green. Every pack in the fleet is ~13× the cliff, so no small fixture could ever have caught this — which is exactly how it survived three earlier test passes.

A measured law retired

The first real boot also made the cold-voice claim checkable, and it is wrong. Empty registry, register only, then time:

request cost
first synthesis of a freshly registered voice 1 883 ms
second 1 283 ms
third 1 226 ms
~110-word request, first 9.4 s
same request again 9.8 s

So the trial's 6.2–8.0 s cold / first audio byte 5 530 ms / later requests 0.5 ms ladder does not reproduce: request time tracks the audio generated, not snapshot state, and 0.5 ms was never an end-to-end number. Warming still stays — for the reason the page did not lead with: the registry is memory-only and re-registering does not warm, so a boot without the hook leaves the façade answering 400 for every name. The saving is ~0.6 s, not five seconds.

Corrected in Frank TTS Engine.md (law 1 retitled to the registry law, real journal line, troubleshooting row replaced by the fault that actually happened), annotated rather than erased in Research/Local TTS and Voice Cloning.md, and Voice Packs.md step 6 stops claiming the hook registers only unseen packs.

Verification

  • bats tests/frank_tts_engine.bats → 28/28; new test red before the fix, green after.
  • Live on the deployed container with the three real packs (fixed script podman cp'd in, which is how the host is warm right now): narrator1 cold=1175ms warm=1874ms, narrator2 1373/1892, narrator3 1816/1487, exit 0, 15.2 s for the whole pass.
  • Full ./tests/run.sh result is in the PR thread (443 + this test).

Why this merge is urgent, not routine

The engine container running on Dankclaw has the fix only in its writable layer (podman cp). Quadlet recreates the container on every unit start, so the next restart — or a reboot — brings back the broken hook and the endpoint comes up with zero voices. Merge + ./scripts/deploy.sh dankclaw --service frank-tts-engine makes it durable; frank-tts (7772) waits behind it.

## The fault The first deploy of `frank-tts-engine` built the image (the `useradd` fix holds — `STEP 3/13` passed, unit `healthy` after 31 s), and then reported: ``` /app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long frank-tts-engine-warm: voice 'narrator1': registration failed (HTTP no response) — the shim will answer 400 for this name … narrator2, narrator3 … frank-tts-engine-warm: 0 voice(s) warm, at least one failed (see the lines above) ``` with `/health` answering 200 the whole time. The hook built `{"name":…,"wav_b64":…}` in a shell variable and handed it to curl as **one argument**. The kernel caps a single argument at `MAX_ARG_STRLEN` = **131,072 bytes**; a shipped `reference.wav` (955 KB) base64's to ~**1.27 MB**, so curl was never executed and the hook reported the absence of a response as an HTTP failure. Fourth silent fault in the voice stack, same species as the three in #14: the failure looked like health. ## Fix Registration body → `mktemp` file, curl reads `@file`; the audio enters through a redirection instead of a command substitution, so the script never holds a 1.3 MB string at all. One file for the pass, reused per voice, removed on exit (container `/tmp`, not host state). ## Test The seam's stub engine is a **real listener** and `curl` is the **real binary**, so the fixture pack (300 KB) trips the actual kernel limit: the test fails with the host's own error before the fix, passes after. Proved red → green. Every pack in the fleet is ~13× the cliff, so no small fixture could ever have caught this — which is exactly how it survived three earlier test passes. ## A measured law retired The first real boot also made the cold-voice claim checkable, and it is wrong. Empty registry, register only, then time: | request | cost | |---|---| | first synthesis of a freshly registered voice | **1 883 ms** | | second | 1 283 ms | | third | 1 226 ms | | ~110-word request, first | 9.4 s | | same request again | 9.8 s | So the trial's `6.2–8.0 s cold / first audio byte 5 530 ms / later requests 0.5 ms` ladder does not reproduce: request time tracks the audio generated, not snapshot state, and 0.5 ms was never an end-to-end number. Warming still stays — for the reason the page did not lead with: the registry is memory-only and re-registering does not warm, so a boot without the hook leaves the façade answering **400 for every name**. The saving is ~0.6 s, not five seconds. Corrected in `Frank TTS Engine.md` (law 1 retitled to the registry law, real journal line, troubleshooting row replaced by the fault that actually happened), annotated rather than erased in `Research/Local TTS and Voice Cloning.md`, and `Voice Packs.md` step 6 stops claiming the hook registers only unseen packs. ## Verification - `bats tests/frank_tts_engine.bats` → **28/28**; new test red before the fix, green after. - Live on the deployed container with the three real packs (fixed script `podman cp`'d in, which is how the host is warm right now): `narrator1 cold=1175ms warm=1874ms`, `narrator2 1373/1892`, `narrator3 1816/1487`, exit 0, 15.2 s for the whole pass. - Full `./tests/run.sh` result is in the PR thread (443 + this test). ## Why this merge is urgent, not routine The engine container running on Dankclaw has the fix only in its **writable layer** (`podman cp`). Quadlet recreates the container on every unit start, so the next restart — or a reboot — brings back the broken hook and the endpoint comes up with zero voices. Merge + `./scripts/deploy.sh dankclaw --service frank-tts-engine` makes it durable; `frank-tts` (7772) waits behind it.
dank added 2 commits 2026-10-18 18:46:59 -07:00
The deploy of the newly built engine image came back `0 voice(s) warm` with
`/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long` above
every line, and the unit was `active` with a green `/health` beside it. The hook
built `{"name":…,"wav_b64":…}` in a shell variable and passed it as ONE curl
argument; the kernel caps a single argument at MAX_ARG_STRLEN = 131,072 bytes,
and the shipped reference clips base64 to about 1.27 MB, so curl was never
executed and the hook reported the absence as "HTTP no response".

The body now goes to a mktemp file and curl reads `@file`; the audio enters
through a redirection instead of a command substitution, so the script never
holds a 1.3 MB string at all. Every pack in the fleet is ~1 MB, i.e. ~13x the
cliff, so this was always going to happen — it took the first real packs to show
it because no fixture was ever that big.

The regression test therefore ships a 300 KB pack against the seam's REAL stub
listener with the REAL curl: it fails with the same kernel error the host did
(`Argument list too long`, then the 400-for-every-name summary), and passes once
the body is a file. Proved red before the fix, green after.
The first real boot gave the chance to measure what warming actually buys, and
the trial's numbers were wrong:

- Empty registry, register only, then time: the FIRST synthesis of a freshly
  registered voice cost 1 883 ms; the next two 1 283 and 1 226 ms. A ~110-word
  request cost 9.4 s and then 9.8 s.
- So the published 6.2–8.0 s cold / first-audio-byte 5 530 ms / later-requests
  0.5 ms ladder does not reproduce, and request time tracks the audio generated
  rather than snapshot state. The 0.5 ms figure is an internal state-restore
  cost, never an end-to-end request time.
- Warming still earns its place for a different reason than the page claimed:
  the registry is memory-only and re-registering does not warm, so a boot without
  the hook leaves the façade answering 400 for every name. The saving is ~0.6 s.

Law 1 of [[Frank TTS Engine]] is re-titled the registry law and carries the
measured numbers; its example journal line is a real one from today; the
troubleshooting row for "first request of the day is slow" — the retired law's
symptom — is replaced by the argument-cap fault that actually happened. The same
ladder in the research brief is annotated rather than erased, and [[Voice Packs]]
step 6 stops claiming the hook registers only packs it has not seen (the engine
replaces a name it already holds).
dank merged commit 0315a62d84 into main 2026-10-18 18:50:01 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#15