33 lines
3.2 KiB
Markdown
33 lines
3.2 KiB
Markdown
---
|
||
name: tts-synth-timeout-cap-run-triage
|
||
description: "Triage a dankvoice \"The operation timed out.\" toast: prove/disprove a qwentts engine cap-run (2048-frame, no EOS) via Dankclaw journals before suspecting the client."
|
||
---
|
||
|
||
# TTS synth-timeout triage
|
||
|
||
Symptom: omp shows `Error: dankvoice: The operation timed out.` — a real 60 s stall, not a client bug. `SYNTH_TIMEOUT_MS = 60_000` (`src/audio-io.ts`) + the notify at `utterance.ts:181` is the ONLY emitter of that text.
|
||
|
||
## Known root cause (chronic as of 2026-10)
|
||
|
||
The pinned `qwentts.cpp tts-server` (frank-tts-engine, Dankclaw, container port 8880) can miss EOS and run to its hard-coded 2048-frame cap: ~164 s of audio from a short sentence, ~90–100 s wall. Measured ~5/day, 68 in 14 days. The engine is a serial FIFO (`QWENTTS_MAX_BATCH=1`), so every narration queued behind a cap-run also times out — the toast comes in bursts. Shim logs `BrokenPipeError` at `wfile.write` (client gone when buffered audio finally ships). No frame-cap/deadline flag exists (`/app/tts-server --help`); the shim can't cancel (engine buffers before writing); `SHIM_TIMEOUT=600` deliberately outwaits it. **Fix belongs in `serverAppConfig` (watchdog restart or upstream feature request), never in dankvoice.**
|
||
|
||
## Procedure
|
||
|
||
1. **Endpoint is alive?** `curl -fsS -o /dev/null -w "%{http_code} %{time_total}s\n" -m 10 http://192.168.0.143:7772/v1/audio/voices` and time one synthesis with the session's actual voice (voice may be set per-directory in `~/.omp/cache/dankvoice.json` `byDir` — read that first).
|
||
2. **Read the journals — from bash ssh, NOT the `ssh://dankclaw/` device.** The ssh:// device lands as a different user (blank) than the ssh config (dank@192.168.0.143) and sees no units, no sockets — false negatives everywhere:
|
||
```bash
|
||
ssh -o BatchMode=yes dankclaw 'journalctl --user -u frank-tts --since "-6h" --no-pager | grep -B1 -A20 BrokenPipeError | tail'
|
||
ssh -o BatchMode=yes dankclaw 'journalctl --user -u frank-tts-engine --since "-6h" --no-pager | grep -E "Generation done : (2048|[89][0-9]{2}) frames"'
|
||
```
|
||
Signature = a facade `BrokenPipeError` whose timestamp is ~60 s after an engine `Admit slot=0`, and/or `Generation done : 2048 frames`. Window-correlate with the engine: `--since "HH:MM" --until "HH:MM"` and skip `[Sample]`/`Generated` noise.
|
||
3. **Chronicity count:** `journalctl --user -u frank-tts-engine --since "-14 days" --no-pager | grep -c "2048 frames"`.
|
||
4. **Verdict:** cap-run confirmed → the client behaved correctly (toast + next utterance speaks). Route the fix to `dank/serverAppConfigs` (`dankclaw/voiceservice/`): host-side watchdog on the journal signature restarting `frank-tts-engine.service`, and/or upstream per-request frame budget.
|
||
|
||
## Traps
|
||
|
||
- `systemctl --user` / `journalctl --user` on Dankclaw require the `dank` user — rootless quadlet units live there.
|
||
- zsh on Dankclaw eats `===` separators and unquoted URLs as globs: quote args or avoid `echo ===`.
|
||
- `tts-server` is not on the exec PATH: use `/app/tts-server`.
|
||
- Roster may contain only cloned voices (narrator1-4 retired) — a 400 on a stale voice name is a separate bug; timeouts are the cap-run.
|
||
- Elevated RTF right after idle (~2 min gap) is iGPU clock ramp, not contention.
|