dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-tts): cloned-voice speech endpoint (phase 1) #4

Merged
dank merged 1 commits from wt/20261017-181502 into main 2026-10-18 14:08:13 -07:00
Conversation 0 Commits 1 Files Changed 5
+941
dank commented 2026-10-18 14:07:25 -07:00
Owner
Copy Link
Copy Source

What lands

frank-tts — the fleet's cloned-voice speech endpoint, phase 1: the façade, its
contract, its tests, its unit. Not deployed (no engine behind it yet).

One commit: feat(frank-tts): cloned-voice speech endpoint — OpenAI-shaped facade, voice packs by name.

The surface

POST /v1/audio/speech with voice=<name> → audio in that voice, where a name
is real exactly when /voices/<name>/reference.wav exists.

  • unknown voice → 400 listing the voices that are; never a silent fallback
    (a wrong-voice endpoint stays wrong for weeks)
  • voice_id accepted as the alias the omp harness sends
  • response_format=mp3 really returns mp3 (ffmpeg re-encode from the engine's wav)
  • oversized input → 414, refused outright, never half-narrated
  • every other route → 404; an unrouted POST does not poison a reused HTTP/1.1 connection
  • /health reports the backend and the resolved voice list (mounted-but-empty
    /voices still reads healthy; a dead app does not)
  • audio bytes, reference audio and synthesized text never reach the log

Two backends behind the one surface, TTS_BACKEND:

backend mechanism
omni-http forward to an OpenAI-shaped upstream (host.containers.internal:8880), carrying the voice as base64 reference_audio; shim re-encodes on the way out
llama-tts-cli run $LLAMA_TTS_BIN once per request with --tts-speaker-file <reference.wav>, nice -n 19, 8 threads — narration must never starve the text endpoints on this box

Naming and placement (Frank's law)

First-class fleet model endpoint → alias-named frank-tts (TTS endpoint, not
ASR), port 7772 in the 7770 frank-dragoon / 7771 frank-zealot family; the
engine upstream keeps 8880 and is not a fleet endpoint. Rootless quadlet:
Restart=always, RestartSec=10, WantedBy=default.target, image pinned by
digest base, Pull=never + localhost/ prefix, Network=dankclaw_default,
/home/dank/voices mounted read-only, uid 10000, no GPU devices (it is a
façade).

Voice-pack law (recorded in scripts/frank-tts.env)

A directory is a pack only with both required members:

  1. reference.wav — 24 kHz mono PCM, 18–25 s clean single-speaker audio (+ the
    16 kHz variant some engines want)
  2. the engine's own speaker artifact, required beside it and never replacing
    the WAV — a voice that cannot be reused by another runtime is not a saved voice

plus manifest.json provenance: source file, offsets (always past the first
10 s of that source — the source-material law), engine, sha256. Enforcement
split is explicit: the shim gates on reference.wav because that is the only
engine-agnostic term; the pack builder is what refuses to emit a pack without
its engine artifact, until a winning engine pins the format.

Tests

tests/frank_tts_unit.bats (12 tests) drives the real app.py over
127.0.0.1 ephemeral ports with a temp VOICES_DIR and stubbed backends — no
Dankclaw, no containers, no GPU; every process it starts is recorded by pid and
killed in teardown. Full suite: 384/384 green (./tests/run.sh, wall 66 s).

Phase 2 (deliberately not here)

  • engine decision + the :8880 upstream (issue #2's VoxCPM2 candidate) and its
    speaker-artifact format
  • pack builder: candidate census is running now (9 candidates / 3 voices →
    ear-pick), then pack assembly with manifest + artifact
  • deployer service mapping, homepage card, Prometheus scrape
  • the models.yml row that lets omp call it: tts dispatches an
    api: openai-speech provider row to POST <baseUrl>/audio/speech with
    {model, input, response_format, voice}, so the tool path speaks this
    endpoint natively. Caveat measured from the harness docs: the live speech.*
    vocalizer is Kokoro-enum-only (12 hard-coded voices, no documented key
    redirecting it to an HTTP endpoint), so "the agent talks in my voice" needs
    either that provider row consumed through the tts path or a
    pi.registerProvider extension — a phase-2 decision, not assumed here
  • open-notebook quadlet + the 3-voice test podcast
# What lands `frank-tts` — the fleet's cloned-voice speech endpoint, phase 1: the façade, its contract, its tests, its unit. **Not deployed** (no engine behind it yet). One commit: `feat(frank-tts): cloned-voice speech endpoint — OpenAI-shaped facade, voice packs by name`. ## The surface `POST /v1/audio/speech` with `voice=<name>` → audio in *that* voice, where a name is real exactly when `/voices/<name>/reference.wav` exists. - unknown voice → **400** listing the voices that are; never a silent fallback (a wrong-voice endpoint stays wrong for weeks) - `voice_id` accepted as the alias the omp harness sends - `response_format=mp3` really returns mp3 (ffmpeg re-encode from the engine's wav) - oversized `input` → **414**, refused outright, never half-narrated - every other route → 404; an unrouted POST does not poison a reused HTTP/1.1 connection - `/health` reports the backend and the resolved voice list (mounted-but-empty `/voices` still reads healthy; a dead app does not) - audio bytes, reference audio and synthesized text never reach the log Two backends behind the one surface, `TTS_BACKEND`: |backend|mechanism| |---|---| |`omni-http`|forward to an OpenAI-shaped upstream (`host.containers.internal:8880`), carrying the voice as base64 `reference_audio`; shim re-encodes on the way out| |`llama-tts-cli`|run `$LLAMA_TTS_BIN` once per request with `--tts-speaker-file <reference.wav>`, `nice -n 19`, 8 threads — narration must never starve the text endpoints on this box| ## Naming and placement (Frank's law) First-class fleet model endpoint → alias-named `frank-tts` (TTS endpoint, not ASR), **port 7772** in the `7770 frank-dragoon` / `7771 frank-zealot` family; the engine upstream keeps 8880 and is not a fleet endpoint. Rootless quadlet: `Restart=always`, `RestartSec=10`, `WantedBy=default.target`, image pinned by digest base, `Pull=never` + `localhost/` prefix, `Network=dankclaw_default`, `/home/dank/voices` mounted **read-only**, uid 10000, no GPU devices (it is a façade). ## Voice-pack law (recorded in `scripts/frank-tts.env`) A directory is a pack only with **both** required members: 1. `reference.wav` — 24 kHz mono PCM, 18–25 s clean single-speaker audio (+ the 16 kHz variant some engines want) 2. **the engine's own speaker artifact**, required beside it and never replacing the WAV — a voice that cannot be reused by another runtime is not a saved voice plus `manifest.json` provenance: source file, offsets (**always past the first 10 s of that source** — the source-material law), engine, sha256. Enforcement split is explicit: the shim gates on `reference.wav` because that is the only engine-agnostic term; the pack builder is what refuses to emit a pack without its engine artifact, until a winning engine pins the format. ## Tests `tests/frank_tts_unit.bats` (12 tests) drives the real `app.py` over 127.0.0.1 ephemeral ports with a temp `VOICES_DIR` and stubbed backends — no Dankclaw, no containers, no GPU; every process it starts is recorded by pid and killed in teardown. Full suite: **384/384 green** (`./tests/run.sh`, wall 66 s). ## Phase 2 (deliberately not here) - engine decision + the `:8880` upstream (issue #2's VoxCPM2 candidate) and its speaker-artifact format - pack builder: candidate census is running now (9 candidates / 3 voices → ear-pick), then pack assembly with manifest + artifact - deployer service mapping, homepage card, Prometheus scrape - the `models.yml` row that lets omp call it: `tts` dispatches an `api: openai-speech` provider row to `POST <baseUrl>/audio/speech` with `{model, input, response_format, voice}`, so the **tool** path speaks this endpoint natively. Caveat measured from the harness docs: the live `speech.*` vocalizer is Kokoro-enum-only (12 hard-coded voices, no documented key redirecting it to an HTTP endpoint), so "the agent talks in my voice" needs either that provider row consumed through the tts path or a `pi.registerProvider` extension — a phase-2 decision, not assumed here - open-notebook quadlet + the 3-voice test podcast
dank added 1 commit 2026-10-18 14:07:25 -07:00
feat(frank-tts): cloned-voice speech endpoint — OpenAI-shaped facade, voice packs by name 31266fb7f3 
Phase 1 of the voice-over goal (Gitea issue #2). One stdlib-only HTTP facade in
front of whichever TTS engine wins the engine decision: POST /v1/audio/speech
with voice=<name> answers audio in THAT voice, where a name is real exactly when
/voices/<name>/reference.wav exists — an unknown name is a 400 listing the
voices that are, never a silent fallback (a wrong-voice endpoint stays wrong for
weeks). voice_id works as the alias the omp harness sends; response_format=mp3
is really mp3 (ffmpeg re-encode); oversized input is refused with 414 instead of
half-narrated; /health reports the backend and the resolved voice list.

Two backends behind the one surface, chosen by TTS_BACKEND: omni-http forwards to
an OpenAI-shaped upstream carrying the voice as base64 reference_audio, and
llama-tts-cli runs a local binary per request with --tts-speaker-file at nice 19
so narration never starves the text endpoints. Both are driven in the seam
through stubs — no GPU, no containers.

Named frank-tts, a first-class fleet model endpoint: port 7772 in the 7770/7771
family, alias naming per the StarCraft law, rootless boot-persistent self-healing
quadlet with a read-only /home/dank/voices mount. Voice-pack law recorded in the
env header: reference.wav PLUS the engine's own speaker artifact (required, so a
voice is reusable in other runtimes), plus manifest.json provenance with offsets
always past the first 10 s of the source file.

Not deployed: the engine decision, the pack builder, deployer/homepage/Prometheus
registration and the omp models.yml runner row are phase 2.
dank added 1 commit 2026-10-18 14:07:26 -07:00
feat(frank-tts): cloned-voice speech endpoint — OpenAI-shaped facade, voice packs by name 31266fb7f3 
Phase 1 of the voice-over goal (Gitea issue #2). One stdlib-only HTTP facade in
front of whichever TTS engine wins the engine decision: POST /v1/audio/speech
with voice=<name> answers audio in THAT voice, where a name is real exactly when
/voices/<name>/reference.wav exists — an unknown name is a 400 listing the
voices that are, never a silent fallback (a wrong-voice endpoint stays wrong for
weeks). voice_id works as the alias the omp harness sends; response_format=mp3
is really mp3 (ffmpeg re-encode); oversized input is refused with 414 instead of
half-narrated; /health reports the backend and the resolved voice list.

Two backends behind the one surface, chosen by TTS_BACKEND: omni-http forwards to
an OpenAI-shaped upstream carrying the voice as base64 reference_audio, and
llama-tts-cli runs a local binary per request with --tts-speaker-file at nice 19
so narration never starves the text endpoints. Both are driven in the seam
through stubs — no GPU, no containers.

Named frank-tts, a first-class fleet model endpoint: port 7772 in the 7770/7771
family, alias naming per the StarCraft law, rootless boot-persistent self-healing
quadlet with a read-only /home/dank/voices mount. Voice-pack law recorded in the
env header: reference.wav PLUS the engine's own speaker artifact (required, so a
voice is reusable in other runtimes), plus manifest.json provenance with offsets
always past the first 10 s of the source file.

Not deployed: the engine decision, the pack builder, deployer/homepage/Prometheus
registration and the omp models.yml runner row are phase 2.
dank merged commit 49c4e198fd into main 2026-10-18 14:08:13 -07:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
human-only
Owner-only decision; agents must not pick this up
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
dank
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#4