Frank
6215f1a6c8
- frank-dragoon: measured the response_format contract. json_schema and json_object answer
schema-valid JSON at temperature 0 and draw a 400 at any sampled temperature, and because
the sampler default is server-side an absent temperature is sampled. The page claimed
"response_format is unsupported", which sends the next agent past a working feature.
- frank-dragoon: new "The NPU lane" section. The pinned 0.16.2 image already ships the
five-program AIE catalog including qwen3-embedding-0.6b, so /v1/embeddings is one env var
plus the accel device, the distro-XRT mount and the fabric-clock hold; embeddinggemma-2
needs 0.18.3+, the stock-kernel-driver memory-budget fix 0.18.8+. Host census recorded:
no /opt/xilinx/xrt, distro XRT 1:2.21.75 with libxrt_driver_xdna, clocks unheld.
- Open Notebook: scriptwriter probe verdict — greedy-only, and open-notebook's model record
is {credential, name, provider, type} with no per-row config, so it cannot ask for greedy;
frank-archon stays the writer. The embedding gap now has a named door instead of "build a
new llama.cpp slot".
- Frank TTS: "the NPU is not a door at all" was too broad. It is not a door for the TTS
family; halogen's precompiled catalog is a door on the same device.
- System Specs: NPU runtime row corrected to the distro XRT actually installed, and the
stale "frank-zealot serves 27B" line retired in favour of [[frank-zealot]] as sole owner.
Tests: ./tests/run.sh, 455 passed.
226 lines
14 KiB
Markdown
226 lines
14 KiB
Markdown
---
|
|
type: reference
|
|
description: "The fleet's cloned-voice speech endpoint and the one voice surface consumers address; wired as the harness speech role."
|
|
tags:
|
|
- endpoint
|
|
- dankclaw
|
|
- tts
|
|
- voice
|
|
- quadlet
|
|
timestamp: 2026-10-19
|
|
---
|
|
|
|
# Frank TTS
|
|
|
|
`frank-tts` is the fleet's only speech endpoint: an OpenAI-shaped
|
|
`POST /v1/audio/speech` on **Dankclaw :7772** that answers `voice=<name>` with
|
|
audio in that cloned voice. Everything else in the voice stack sits behind it —
|
|
the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers
|
|
never learn a path.
|
|
|
|
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
|
|
both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
|
|
the same change. Probes on the deployed endpoint: `/health` answers
|
|
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`
|
|
and two different voices rendered the same sentence into **different** bytes
|
|
(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
|
|
pass-through of one default. The operator harness routes its `speech` role here;
|
|
see [[#The harness speech role]]. What still waits is [[Open Notebook]]: its
|
|
render needs the app installed, not this endpoint.
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
|
|
| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
|
|
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only. The request's `model` field is **ignored** (measured 2026-10-19: `voxcpm2` and `tts-1` both synthesize the same way) — the voice comes solely from `voice`, so a consumer that must name a model may name it anything |
|
|
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
|
|
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
|
|
| Voice packs | `/home/dank/voices` mounted read-only at `/voices` — host state, never repo state |
|
|
| Upstream | `http://frank-tts-engine:8880` (`scripts/frank-tts.env`, `TTS_UPSTREAM`) |
|
|
| Default voice | `DEFAULT_VOICE` (`scripts/frank-tts.env`) answers a request that names none; the journal marks that line `(default)`, and an **unknown** name is still a 400 |
|
|
| Rollback | `TTS_BACKEND=llama-tts-cli` runs a CLI binary per request and needs no upstream at all |
|
|
|
|
## Naming a voice
|
|
|
|
A voice **is** a directory name. `voice=narrator1` resolves to
|
|
`/voices/narrator1/reference.wav`, and that file — not a manifest, not a config
|
|
entry — is what makes the name real. omp's TTS device may send `voice_id`; the
|
|
shim accepts either. A call that sends **neither** — the harness's normal shape,
|
|
because no harness setting carries a voice for this transport — speaks
|
|
`$DEFAULT_VOICE`; empty default makes it a 400 again.
|
|
|
|
The usable shape is `^[a-z0-9][a-z0-9._-]*$`: lowercase, starting with a letter
|
|
or digit, then letters, digits, `.`, `_`, `-`, at most 64 characters. No slash,
|
|
no `..`, no space, no uppercase. The lowercase requirement is where two rules
|
|
meet: `app.py` would accept `[A-Za-z0-9…]`, but `tts-server` **lowercases
|
|
names**, so a pack called `Narrator1` resolves through the façade and not
|
|
through the engine. The boot warmer therefore refuses a mixed-case directory
|
|
loudly instead of letting it half-work.
|
|
|
|
Names are user-visible: an unknown voice comes back as
|
|
`400 unknown voice 'x'; available voices: …`, and that list is the honest
|
|
census. The podcast speakers are `narrator1`, `narrator2`, `narrator3` —
|
|
[[Open Notebook]] is configured with exactly those three, so renaming them
|
|
breaks episode rendering, not just speech. A personal voice gets the person's
|
|
name, lowercase.
|
|
|
|
## Voice packs
|
|
|
|
One directory per voice under `/home/dank/voices`:
|
|
|
|
```
|
|
reference.wav 24 kHz mono PCM, 18-25 s of clean single-speaker audio
|
|
reference.spk engine speaker embedding (8 192 B)
|
|
reference.rvq reference codec codes (22 B per 12.5 Hz frame)
|
|
manifest.json source file, offsets, engine, sha256
|
|
```
|
|
|
|
`reference.wav` is the engine-agnostic term the façade gates on; the other two
|
|
are what makes a voice reusable by this engine, and they come from the pack
|
|
builder ([[Voice Packs]]) — never by hand, never fabricated from the WAV. Two
|
|
laws apply to the
|
|
source material: offsets always start **past the first 10 seconds** of the
|
|
sample, and packs are host state that stays out of this repo. Because the
|
|
container runs as uid 10000, files must be world-readable (`644`/`755`), and the
|
|
directory must exist before the first deploy: rootless podman auto-creates a
|
|
missing bind source **root-owned**, which reads as "no voices" rather than as an
|
|
error.
|
|
|
|
Adding a voice to a running stack is a two-step thing — drop the pack, then
|
|
register **and warm** it with
|
|
`podman exec frank-tts-engine /app/warm-voices.sh` — because registering alone
|
|
does not warm it. The reason is the cold-voice law in [[Frank TTS Engine]], and
|
|
the procedure with its proof (the second request under a second) is the
|
|
`frank-tts-add-cloned-voice` skill.
|
|
|
|
## Why the upstream is a container name
|
|
|
|
`scripts/frank-tts.env` used to point at `http://host.containers.internal:8880`.
|
|
That cannot work, and it is the trap this shape exists to avoid: a bridge
|
|
container reaches the host's network namespace, netavark DNATs **published**
|
|
ports only, and an engine on the host's loopback is published to nobody — the
|
|
same shape that 502'd hermes ([[Container Network Namespaces]]). So both units
|
|
attach to one committed Quadlet network and the façade dials the engine's
|
|
container DNS name.
|
|
|
|
The consequence worth stating before an operator meets it: **the engine is not
|
|
reachable from the host at all** — not over the LAN, not over loopback. Its
|
|
health, voice registry and logs are observable only through
|
|
`podman exec frank-tts-engine …` and `systemctl --user status`. And the façade's
|
|
own `/health` answers 200 with the backend name and the voice list it resolved
|
|
without dialing the engine, so "façade healthy, engine down" is a real state: it
|
|
shows up as a failed synthesis, never as a down probe.
|
|
|
|
## Where the engine computes
|
|
|
|
The **gfx1151 iGPU**, as a co-tenant beside `frank-dragoon`: the Vulkan build of
|
|
`qwentts.cpp` with `--device Vulkan0`, and `/dev/dri` handed to the engine container. That is
|
|
the shape since 2026-10-19. The flag, the device name, the render-node trap on this hybrid box
|
|
and the one-line CPU rollback are owned by [[Frank TTS Engine]] — read that page before
|
|
touching either tenant, because a change to one is a change to both. Co-tenancy was gated with
|
|
`frank-dragoon` live on the card (its first token and decode rates unchanged with a synthesis
|
|
in flight); both gate recipes are §5 of `report/tts-npu-spike-verdict-2026-10-19.md`.
|
|
|
|
**The NPU is a door only for a model the vendor has already compiled for it.** XDNA2 runs nothing but
|
|
precompiled AIE partition programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the
|
|
ioctls, then declines this model family's graph without emitting a single log line. That is why TTS stays on
|
|
the GPU — and it is not a general veto on the device: halogen ships its own precompiled catalog (embeddings,
|
|
decisions, rerankers) that does run there, which is what [[frank-dragoon]]'s NPU-lane section is about. The
|
|
measured record and the model-side contract live in `report/` at the repo root
|
|
(`tts-npu-spike-verdict-2026-10-19.md`).
|
|
|
|
## Files
|
|
|
|
| Path | What |
|
|
| --- | --- |
|
|
| `dankclaw/voiceservice/app.py` | the shim: stdlib only — resolves the voice, forwards, re-encodes mp3 |
|
|
| `dankclaw/voiceservice/Containerfile` | python 3.12-alpine pinned by digest + ffmpeg, uid 10000 |
|
|
| `dankclaw/voiceservice/quadlet/frank-tts.container` | the unit: two networks, `Restart=always`, `HealthCmd`, `[Install]` |
|
|
| `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the shared bridge, named after the engine unit and attached by both |
|
|
| `scripts/frank-tts.env` | backend, upstream, port, limits, and the default voice — paths and numbers, no credentials |
|
|
| `tests/frank_tts_unit.bats` | the shim against stub engines over real sockets |
|
|
| `tests/frank_tts_engine.bats` | the pair's wiring: generator output, planner, homepage, deployer |
|
|
|
|
## Deploying it
|
|
|
|
`./scripts/deploy.sh dankclaw --service frank-tts` (or `--service frank-tts-engine --service frank-tts` for the pair):
|
|
builds the façade image, installs **both** Quadlet units — including the
|
|
engine's `.network` file, because a referenced `Network=` file the generator
|
|
cannot find means the façade unit is not produced at all — creates
|
|
`/home/dank/voices` as the deploy user, restarts the unit (whose `Wants=` and
|
|
`After=` pull the engine and its warm pass ahead of it) and gates on podman's
|
|
recorded health. It never runs `systemctl --user enable`; boot persistence is
|
|
the `[Install]` section in each unit. Order matters in a full deploy:
|
|
[[Frank TTS Engine]] is planned before this, and both before [[Open Notebook]].
|
|
|
|
## Not scraped, and why
|
|
|
|
`shared/prometheus/` gets **no** frank-tts job, and that is a considered
|
|
absence. The façade exports no `/metrics` — `app.py` serves `/health` plus the
|
|
speech route only — and this Prometheus 3 install refuses non-metrics content
|
|
types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two
|
|
`services` targets) sit **down** with
|
|
`received unsupported Content-Type … and no fallback_scrape_protocol specified`.
|
|
A `/health` probe job would therefore be structurally never-green, and because
|
|
`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
|
|
lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
|
|
Signal every few minutes for an endpoint that is fine. The engine cannot be
|
|
scraped at all: it publishes nothing. Status lives in `systemctl --user status`
|
|
`podman healthcheck run`, the two unit journals, and the homepage card, which
|
|
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
|
|
2026-10-18 deploy.
|
|
|
|
## The harness speech role
|
|
|
|
The operator harness reaches this endpoint through
|
|
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`
|
|
`baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api
|
|
`openai-speech` on the **model** row (naming a runner api at *provider* level
|
|
fails the whole file's validation and disables every custom provider, which is
|
|
how the fleet's chat endpoints disappear) — plus `config.yml`
|
|
`modelRoles.speech: frank-tts (dankclaw)/frank-tts`. Four laws, read out of the
|
|
harness bundle and measured the day of the flip:
|
|
|
|
- **The only voice the harness can send is the `voice_id` argument of the call.**
|
|
The transport builds `{model, input, response_format, voice?}` and adds `voice`
|
|
**only** when the caller passed `voice_id`; no setting feeds it.
|
|
`tts.localVoice` is the *local Kokoro backend's* voice (it is what answered
|
|
`voice=kokoro/bm_george, backend=local` in the control test), and
|
|
`speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list,
|
|
and it never reaches an HTTP request. So a harness call arrives voice-less by
|
|
default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared
|
|
pack answers a voice-less request and the journal marks that line `(default)`
|
|
while an **unknown** name still 400s with the list that exists.
|
|
- **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not
|
|
select a pack; it hands the vocalizer a value outside its enum. (The first
|
|
draft of this section said the opposite and was reverted the same day, after a
|
|
fresh session's call came back `unknown voice None`.)
|
|
- **There is no silent fallback.** An explicitly configured `modelRoles.speech`
|
|
yields exactly one candidate, so a dead 7772 fails the speech device with
|
|
*"No available speech model matches the speech role"* instead of quietly
|
|
degrading to Kokoro. That is the price of `auth: none` (which makes a dead
|
|
endpoint read as available) and it is the wanted shape: a narration job should
|
|
fail loudly rather than ship a voice Frank never asked for.
|
|
- **A session older than the models.yml row cannot see it.** `models.yml` loads
|
|
at process start while `config.yml` is read live, so a long-lived session
|
|
carrying the flipped role fails exactly like a dead endpoint while a fresh
|
|
process succeeds in the same minute. Measured 2026-10-18: the open session
|
|
failed twice (provider/id form, then the bare id), `local/kokoro` succeeded in
|
|
that same session, and a fresh `omp -p --tools tts` returned `Saved 180524
|
|
bytes (model=frank-tts (dankclaw)/frank-tts, voice=narrator1, codec=wav)`
|
|
while the façade logged the matching `speech voice=narrator1 format=wav
|
|
chars=51 -> 180524 bytes` from the podman bridge. Rule the stale registry out
|
|
before blaming the service.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Cause |
|
|
| --- | --- |
|
|
| `400 unknown voice 'x'; available voices: …` | the name is not a pack: case, spelling, missing `reference.wav`, or an empty/root-owned `/home/dank/voices` |
|
|
| `(none: no voice pack with a reference.wav in /voices)` | the mount, not the voices: `podman inspect frank-tts --format '{{json .Mounts}}'`, then the directory owner |
|
|
| Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
|
|
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
|
|
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
|
|
| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
|
|
---
|