chore: retire dankvoice drop-in for the dank/dankvoice package
The extension's full behaviour now lives in dank/dankvoice (41/41 black-box suite against the built artifact; this branch's 26 drop-in tests are a strict subset, names ported 1:1). Delete the drop-in and its bats suite; repoint omp-docs/core/dankvoice.md at the package, keeping only fleet wiring (frank-tts + @tiny resolution, the enable_thinking hybrid law, voice/roster contract) and the cutover procedure. kb-lint 0 violations; bats tests/ 45/45. Refs #34
This commit is contained in:
Frank
+55
-223
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: dankvoice
|
||||
description: omp extension that speaks assistant output with the fleet's cloned voices through frank-tts.
|
||||
description: Voice output ships as the dank/dankvoice package; this page keeps only the fleet integration facts and the cutover procedure.
|
||||
tags:
|
||||
- omp
|
||||
- extension
|
||||
@@ -15,244 +15,76 @@ timestamp: 2026-10-19
|
||||
|
||||
# dankvoice
|
||||
|
||||
`agent/extensions/dankvoice.ts` adds cloned-voice speech to the interactive
|
||||
harness: `/say` opens the options menu, `/say <text>` speaks through the
|
||||
speakers and `/say on` narrates the assistant's thinking and final answer
|
||||
after every turn. The words are first rewritten into natural spoken prose by
|
||||
the **@tiny** role model (exactly what the built-in vocalizer's
|
||||
`speech.enhanced` does), and everything is served by **frank-tts**
|
||||
(Dankclaw :7772, the `speech` model role in [agent/models.yml](/core/index.md)).
|
||||
Voice output lives in the **`dank/dankvoice` package**:
|
||||
`pi install git:http://192.168.0.143:3000/dank/dankvoice.git`. This page owns
|
||||
only fleet integration and the cutover. The package README
|
||||
(<http://192.168.0.143:3000/dank/dankvoice>) owns the `/say` command
|
||||
reference, the settings/env knob list, the segmentation state machine, the
|
||||
enhancer port, and mood control — never duplicate those facts here.
|
||||
|
||||
## Why it exists
|
||||
## Fleet wiring
|
||||
|
||||
Nothing built into the harness can pronounce Frank's cloned voices.
|
||||
* **TTS endpoint** — resolved from the provider block whose model row carries
|
||||
`api: openai-speech` in [agent/models.yml](/core/index.md):
|
||||
**frank-tts** on Dankclaw `:7772`. LAN-only — no Caddy vhost fronts it.
|
||||
* **Narration humanizer** — the harness `@tiny` role
|
||||
(`modelRoles.tiny` in `agent/config.yml`), on this fleet **frank-zealot**
|
||||
(Dankclaw `:7771`).
|
||||
* **Hybrid-endpoint trap (load-bearing)** — a raw `fetch` to a
|
||||
thinking-by-default model spends the token budget on `reasoning_content`
|
||||
and answers `content: ""`. The package pins
|
||||
`chat_template_kwargs: { enable_thinking: false }` on the rewrite call; do
|
||||
not remove it. On a hybrid endpoint, empty content beside non-empty
|
||||
`reasoning_content` means the flag was ignored — that is a failed rewrite
|
||||
(fall back to the mechanical text), never the legitimate "nothing to say".
|
||||
* **Voice contract** — omitting `voice` is legal: frank-tts fills its
|
||||
`DEFAULT_VOICE` (2026-10-18 voice-pack work). `GET {base}/v1/audio/voices`
|
||||
answers a bare JSON array; an unknown voice answers 400 with
|
||||
`available_voices`, which the package uses as roster fallback. On this shim
|
||||
`GET /v1/voices` and `GET /v1/models` are 404.
|
||||
|
||||
| Surface | Engine | Verdict |
|
||||
|---|---|---|
|
||||
| TUI speak-aloud (`speech.enabled`, `speech.voice`) | bundled Kokoro ONNX worker, `synthesizeStream(voice, speed, signal)` | Kokoro voice enum only; never consults `models.yml` or the speech role |
|
||||
| `omp say [TEXT]` | same local backend (`--model` = *local* model key) | `omp say --voice narrator1 "test"` → `error: Expected --voice to be one of: af_heart, af_bella, af_nicole, af_kore, af_sarah, am_michael, am_fenrir, am_puck, bf_emma, bm_george, bm_fable` (verified 2026-10-18) |
|
||||
| `tts` tool / `xd://tts` device (`modelRoles.speech`) | frank-tts over OpenAI `/v1/audio/speech` | cloned voices work, but only as a **file** — nothing plays them |
|
||||
## Cutover (Refs #34)
|
||||
|
||||
`dankvoice` is not a patch to that vocalizer (harness code, not ours); it is a
|
||||
second transport that happens to speak, so Kokoro stays available for quick
|
||||
local speech and the cloned voices carry the real narration.
|
||||
The `agent/extensions/dankvoice.ts` drop-in and its 26-test bats suite are
|
||||
retired here (the package suite is a strict superset; test names ported 1:1).
|
||||
Per box:
|
||||
|
||||
## Commands
|
||||
1. `pi install git:http://192.168.0.143:3000/dank/dankvoice.git`
|
||||
2. `/say status`, then "Speak a test phrase" from the `/say` menu — the
|
||||
audible proof a silent session needs.
|
||||
3. Pull `main` (the drop-in disappears with it) and restart the session.
|
||||
|
||||
| Command | Effect |
|
||||
|---|---|
|
||||
| `/say` \| `/say menu` | options menu (`ctx.ui.select`): auto-speak toggle, voice, output device, player, spoken cap, thinking-narration toggle, humanize toggle, test phrase |
|
||||
| `/say <text>` | synthesize once and play; interrupts whatever is playing |
|
||||
| `/say on` \| `/say off` | arm/disarm auto-speak of the thinking blocks and final text at `agent_end`; the footer shows `🔊 <voice>` while armed |
|
||||
| `/say voice <name>` | pick a voice; an empty name defers to the server's own default |
|
||||
| `/say voices` | list what frank-tts reports |
|
||||
| `/say stop` | cut the current utterance |
|
||||
| `/say status` \| `/say help` | endpoint, voice, player, device, cap, thinking, humanize |
|
||||
Menu choices survive the cutover: both transports persist to the same
|
||||
gitignored settings file, `~/.omp/cache/dankvoice.json`
|
||||
(`DANKVOICE_CONFIG` relocates it).
|
||||
|
||||
Menu choices persist to `~/.omp/cache/dankvoice.json` (runtime state, never
|
||||
tracked); the environment overrides the file, the file overrides defaults.
|
||||
"Speak a test phrase" synthesizes a short line through the current voice,
|
||||
player and device — the audible proof whenever a session "has say on" but
|
||||
stays silent.
|
||||
## Why not the built-in vocalizer
|
||||
|
||||
Auto-speak is skipped in `print`/`json` mode so benches and pipelines stay
|
||||
silent. Each spoken unit — thinking first when narrated, then the answer —
|
||||
is first rewritten into spoken prose by the @tiny model; when the enhancer
|
||||
(below) declines, the text goes to segmentation as-is, where fenced code and
|
||||
tables vanish, links collapse to their label and bullets and table pipes
|
||||
disappear. A long answer is never one monolithic synthesis request; see
|
||||
[Segmentation](#segmentation) for how utterances are cut and played.
|
||||
|
||||
## Segmentation
|
||||
|
||||
Utterances are cut by a state machine ported from the harness's own
|
||||
speak-aloud vocalizer (the minified `fHe` class in the agent bundle's
|
||||
`dist/cli.js`), kept deliberately in lockstep with the stock path:
|
||||
|
||||
* A strong boundary is a run of `[.!?…]` followed by whitespace — so
|
||||
`!!!!!` and `...` are one boundary, never a pile of cuts.
|
||||
* Boundaries after `e.g.`, `etc.`, `Mr.` and friends, or inside inline
|
||||
code, don't count; decimals (`3.5`, `1,5`) survive because a boundary
|
||||
requires trailing whitespace.
|
||||
* Line starts are classified: table rows and fenced code are swallowed
|
||||
whole (their content is never read aloud), heading and bullet markers
|
||||
vanish, ordered-list markers speak their number ("1, …"), quote markers
|
||||
strip.
|
||||
* Per segment, markdown is cleaned: links collapse to their label, URLs
|
||||
to their host, deep paths to their last segment; emphasis markup dies.
|
||||
* Segments aim for one sentence; run-ons break at commas or word wraps
|
||||
before 280 characters.
|
||||
|
||||
Playback is strictly sequential: segment N+1 is synthesized only after
|
||||
segment N's player process exited, so audio never stacks. The chain runs
|
||||
detached from `agent_end` (fire-and-forget) and may legitimately outlive
|
||||
the harness's 30 s handler timeout — the old awaiting design produced a
|
||||
"handler timed out after 30000ms" toast on long answers. A new utterance,
|
||||
`/say stop`, or disarming auto-speak retires the live chain through a run
|
||||
token: the current player is killed and pending segments are skipped.
|
||||
`DANKVOICE_MAX` is a per-answer character budget: the leading whole
|
||||
segments that fit are spoken, the rest are dropped — the thinking unit
|
||||
draws on it before the answer does.
|
||||
|
||||
## Endpoint resolution
|
||||
|
||||
The base URL is **read from `agent/models.yml`**, from the provider block whose
|
||||
model row carries `api: openai-speech` — the same row the `speech` role
|
||||
resolves, so no host address and no credential appears in this file. From
|
||||
`baseUrl: http://192.168.0.143:7772/v1` it derives
|
||||
`POST …/v1/audio/speech`, `GET …/v1/audio/voices` and `GET …/health`.
|
||||
|
||||
| Environment | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `DANKVOICE_URL` | models.yml | speech base URL (`…/v1`), bypasses the YAML read |
|
||||
| `DANKVOICE_MODEL` | the id in that block (`frank-tts`) | model named in the request body |
|
||||
| `DANKVOICE_CONFIG` | `~/.omp/cache/dankvoice.json` | persisted menu choices (voice, player, sink, cap, auto) |
|
||||
| `DANKVOICE_VOICE` | empty (server default) | default voice name |
|
||||
| `DANKVOICE_AUTO=1` | off | start with auto-speak armed |
|
||||
| `DANKVOICE_MAX` | `700` | spoken-character budget per answer (whole leading segments, shared by the thinking and answer units) |
|
||||
| `DANKVOICE_PLAYER` | detect | player binary, skipping detection |
|
||||
| `DANKVOICE_MODELS` | `~/.omp/agent/models.yml` | YAML to read the endpoint from |
|
||||
| `DANKVOICE_REWRITE_URL` | the @tiny role | chat base URL (`…/v1`) for the enhancer, bypassing role resolution |
|
||||
| `DANKVOICE_REWRITE_MODEL` | the @tiny role's model | chat model named in the rewrite body |
|
||||
| `DANKVOICE_REWRITE=0` | enhancer on when a route resolves | speak mechanical text |
|
||||
| `DANKVOICE_THINKING` | settings file, default on | `0`/`1` force thinking narration off/on |
|
||||
| `DANKVOICE_OMPCONFIG` | `~/.omp/agent/config.yml` | YAML to read the `modelRoles.tiny` role from |
|
||||
|
||||
Player detection order is `pw-play` → `paplay` → `ffplay` (PipeWire first:
|
||||
the workstations run WirePlumber). A saved output device routes the player
|
||||
explicitly — `pw-play --target <sink>` or `paplay --device=<sink>`; ffplay
|
||||
has no device concept and ignores it — while an unset device leaves the
|
||||
player's own default. Sinks are enumerated with `wpctl list audio sinks`;
|
||||
if wireplumber is unreachable the menu offers only "system default". Each
|
||||
utterance is written to `$TMPDIR/dankvoice-*/say-<ms>.wav` and the file plus
|
||||
its directory are removed when the player process exits.
|
||||
|
||||
## Humanized narration (the @tiny role)
|
||||
|
||||
The built-in vocalizer never reads answers aloud verbatim when
|
||||
`speech.enhanced` is on: a small model rewrites each utterance into words a
|
||||
person would say. `dankvoice` ports that enhancer — same system prompt
|
||||
(verbatim from the bundle), same budgets (1536 max tokens, 6 s timeout,
|
||||
4000-char input window), same contract (a failed rewrite falls back to the
|
||||
mechanically cleaned text; an empty rewrite skips the unit as "nothing to
|
||||
say"). The target model is the harness's `@tiny` role, resolved the way the
|
||||
harness resolves it: `modelRoles.tiny` in `agent/config.yml` reads
|
||||
`provider (host)/model:effort`; the provider block in `agent/models.yml`
|
||||
supplies the `baseUrl`; the request is `POST {base}/chat/completions`. On
|
||||
this fleet that is **frank-zealot** (`llamacpp (dankclaw-3090)`,
|
||||
`:7771`) — the 6 s abort keeps a slow tiny model off the critical path.
|
||||
|
||||
The request body pins `chat_template_kwargs: { enable_thinking: false }`,
|
||||
and that field is load-bearing: the harness injects it for managed roles,
|
||||
but a raw `fetch` gets the model's default — thinking on — so the Qwen35
|
||||
hybrid spends the budget on `reasoning_content` and answers `content: ""`
|
||||
(measured: 6.9 s / 6893 reasoning chars / empty content without the flag;
|
||||
0.55 s / 369 clean chars with it). For a hybrid endpoint, empty content
|
||||
beside non-empty `reasoning_content` means the flag was ignored — a failed
|
||||
rewrite (mechanical fallback), never the legitimate "nothing to say".
|
||||
|
||||
Thinking blocks ride the same path: at `agent_end` the last assistant
|
||||
message's true `thinking` parts are narrated (rewritten first, as their own
|
||||
unit, before the answer), while blocks flagged `narration: true` — which
|
||||
providers use to address the user through the thinking channel — count as
|
||||
answer text. The menu, `/say status`, and `DANKVOICE_THINKING` toggle it;
|
||||
the choice persists in the settings file.
|
||||
|
||||
## Verified facts (2026-10-18, extended 2026-10-19)
|
||||
|
||||
* **Extension discovery** — any `.ts` file in `~/.omp/agent/extensions/` loads
|
||||
at startup: a probe file there printed `loaded 2026-10-19T02:36:55Z` from a
|
||||
fresh `omp -p` run. `omp --help` documents `--no-extensions` (discovery off,
|
||||
explicit `-e/--extension` and `--hook` still load) and `plugin install|link`.
|
||||
* **Endpoint** — resolving the live `agent/models.yml` yields
|
||||
`speechUrl http://192.168.0.143:7772/v1/audio/speech`,
|
||||
`voicesUrl …/v1/audio/voices`, `healthUrl …/health`, `model frank-tts`.
|
||||
* **Roster** — `GET /v1/audio/voices` answers the bare JSON array
|
||||
`["narrator1","narrator2","narrator3"]`; `/health` answers
|
||||
`{"status":"ok","backend":"omni-http","voices":[…]}`. `GET /v1/voices` and
|
||||
`GET /v1/models` are **404** on this shim, and an unknown voice answers
|
||||
`400 unknown voice 'zzz'; available voices: …` with `available_voices` in the
|
||||
body — which is the fallback the extension uses if the roster route ever
|
||||
disappears.
|
||||
* **Omitting `voice` is legal** — frank-tts fills its `DEFAULT_VOICE` since the
|
||||
2026-10-18 voice-pack work, so an unset voice synthesizes instead of failing.
|
||||
* **Speech** — `/say "dankvoice live smoke test"` produced a 99,884 B / 2.08 s
|
||||
WAV and `/say voice narrator2` + `/say "cloned voice check"` a 69,164 B /
|
||||
1.44 s WAV (durations from `ffprobe`), each handed to the player; temp
|
||||
directories were gone afterwards.
|
||||
* **Auto-speak** — verified in a real TUI session: `/say on` armed the footer
|
||||
(`🔊 narrator1`), the next finished turn synthesized and played the answer
|
||||
(1.20 s WAV), and an unknown voice name answers with the roster instead of
|
||||
speaking. An `agent_end` carrying `willContinue` (the harness scheduled an
|
||||
auto-retry) is never narrated — only the terminal settle speaks.
|
||||
* **Menu** — verified via driver scenarios 2026-10-18 (Refs #23): the menu
|
||||
lists auto-speak/voice/device/player/cap/test-phrase; a voice chosen in the
|
||||
menu is used by later utterances and survives a fresh load via the settings
|
||||
file; a device chosen in the menu appears in the player invocation as
|
||||
`--target <sink>`.
|
||||
* **Segmented playback** — driver scenarios 2026-10-19: "First stop here.
|
||||
Second one goes!!! Really? Third ... done" produced exactly 3 sequential
|
||||
requests and 3 plays; "Model 3.5 ships today. Version 1,5 too." produced
|
||||
exactly 2 (decimals never split); the markdown fixture's fenced code and
|
||||
URL never reached the fixture endpoint; with `DANKVOICE_MAX=60` only the
|
||||
leading segment was spoken.
|
||||
* **Humanize + thinking (2026-10-19, Refs #27)** — driver scenarios cover the
|
||||
rewrite round-trip (fixture chat route), the 404 → mechanical fallback,
|
||||
thinking narrated as its own unit before the answer, `DANKVOICE_THINKING=0`
|
||||
silencing it, and `narration: true` spoken once as the answer; the endpoint
|
||||
status line shows the resolved humanize model from the @tiny role. Live
|
||||
smoke against the real services: a markdown `/say` became 3 sequential
|
||||
prose segments (mechanical cleanup of that text yields at most 2), and a
|
||||
thinking+answer `agent_end` played thinking first. 22 green in
|
||||
`tests/dankvoice.bats`.
|
||||
* **Thinking-off fix (2026-10-19, Refs #32)** — replaying the exact rewrite
|
||||
body against live frank-zealot: without `chat_template_kwargs.
|
||||
enable_thinking: false` the reply arrives as 6893 `reasoning_content`
|
||||
characters with `content: ""` in 6.9 s (twice the 6 s abort); with it,
|
||||
369 clean characters in 0.55 s. Driver fixtures now pin the flag
|
||||
(`thinkoff=false` on the recorded chat call), turn reasoning-only replies
|
||||
into mechanical speech instead of silence, and time the menu's test
|
||||
phrase (~0 ms handler). 26 green in `tests/dankvoice.bats`.
|
||||
The harness speak-aloud path (`speech.enabled` / `speech.voice`, `omp say`)
|
||||
synthesizes with bundled Kokoro ONNX against a **compile-time voice enum** —
|
||||
cloned voices can never reach it, and no setting routes it to frank-tts.
|
||||
`dankvoice` is a second transport beside it, not a patch: Kokoro stays for
|
||||
quick local speech, the package carries cloned-voice narration. The `tts`
|
||||
tool / `xd://tts` device reach frank-tts too, but produce files, not audio
|
||||
playback.
|
||||
|
||||
## Limits
|
||||
|
||||
* The TUI's own speak-aloud feature still narrates in Kokoro: its voice list is
|
||||
a compile-time enum in the harness bundle, and no setting routes it to
|
||||
frank-tts. Cloned voices live only on this file-producing/speaking path and
|
||||
on the `tts` tool. See [Handy](/tools/handy.md) for the STT half.
|
||||
* `agent_end` narrates thinking blocks and the final message only — mid-turn
|
||||
tool chatter is never spoken. Thinking covers the last assistant message;
|
||||
tool-loop gaps in earlier messages are skipped by design.
|
||||
* Humanized narration is serial: audio starts only after the @tiny rewrite
|
||||
returns (≤ 6 s per unit, then frank-tts synthesis). Say `/say off`-grade
|
||||
latency is the price of prose; `DANKVOICE_REWRITE=0` trades it back for
|
||||
instant mechanical speech.
|
||||
* frank-tts is LAN-only: no Caddy vhost fronts :7772, so this extension is
|
||||
usable on the home network.
|
||||
* Deployment, packs and the voice registry are owned by the
|
||||
`serverAppConfigs` repo (`serverAppConfigs-docs/Dankclaw/Frank TTS.md`), not
|
||||
by this one.
|
||||
* Auto-speak narrates thinking and the final message only — mid-turn tool
|
||||
chatter is never spoken; skipped in `print`/`json` modes so benches stay
|
||||
silent.
|
||||
* frank-tts deployment, voice packs and the registry are owned by the
|
||||
`serverAppConfigs` repo (`serverAppConfigs-docs/Dankclaw/Frank TTS.md`).
|
||||
|
||||
## Citations
|
||||
|
||||
* Live probes on the workstation, 2026-10-18: `omp say --voice narrator1 …`,
|
||||
`curl http://192.168.0.143:7772/health`, `/v1/audio/voices`,
|
||||
`/v1/audio/…/speech` (invalid voice), `ffprobe` on the produced WAVs.
|
||||
* `omp --help` (extension discovery flags, `say`, `plugin`) and
|
||||
`@earendil-works/pi-coding-agent` `dist/core/extensions/types.d.ts`
|
||||
(`ExtensionAPI.on/registerCommand`, `ExtensionMode = "tui" | "rpc" | "json" | "print"`).
|
||||
* The stock speak-aloud implementation — vocalizer pipeline `I6o`, utterance
|
||||
segmenter `fHe` and speech enhancer `Qyr` with its prompt `T6o` — in
|
||||
`@oh-my-pi/pi-coding-agent`'s `dist/cli.js`, read off the minified bundle
|
||||
2026-10-19: dankvoice ports the mechanical and the enhanced halves, so an
|
||||
LLM rewrite does happen before frank-tts.
|
||||
* Gitea issue #21 on `dank/command-center` (the work item this page ships with).
|
||||
* Gitea issue #23 on `dank/command-center` (the options-menu work item).
|
||||
* Gitea issue #27 on `dank/command-center` (the enhanced-narration work item).
|
||||
* Gitea issue #32 on `dank/command-center` (the latency/silence fix work item).
|
||||
* Package repo `dank/dankvoice` (README, `docs/decisions/`) — behaviour and
|
||||
knob truth.
|
||||
* Gitea issues #21, #23, #27, #32 (drop-in history) and #34 (this cutover)
|
||||
on `dank/command-center`.
|
||||
* Live endpoint probes 2026-10-18/19 documented in those issues.
|
||||
|
||||
## See Also
|
||||
|
||||
* [Core Documentation](/core/index.md) — the live harness configuration.
|
||||
* [Handy](/tools/handy.md) — offline speech-to-text, the input half of voice.
|
||||
* `serverAppConfigs-docs/Dankclaw/Frank TTS.md` — the frank-tts service runbook.
|
||||
|
||||
@@ -12,7 +12,7 @@ These files define the structural conventions that govern the agent system itsel
|
||||
|
||||
* [AGENTS.md Format Specification](agents-md-format.md) — Canonical structure and conventions for the system-level instruction file that governs agent behavior.
|
||||
* [Pi Settings System](settings.md) — pi-era `settings.json` record (archive): providers, models, packages, subagent overrides.
|
||||
* [dankvoice](dankvoice.md) — Cloned-voice vocalization extension: `/say` and auto-speak over frank-tts, endpoint read from `agent/models.yml`.
|
||||
* [dankvoice](dankvoice.md) — Voice output is the `dank/dankvoice` package (`pi install git:`); this page keeps fleet wiring and the cutover procedure.
|
||||
* [OMP Status Line Customization](statusline.md) — How presets, composer shapes, and segment arrays actually interact; the fleet's tuned bar. (moved from /pi/, 2026-10-19)
|
||||
* [OMP Subagent Limits & Fan-out Policy](subagent-limits.md) — Live caps: concurrency 2, no nested spawns (the `maxRecursionDepth: 0` trap), per-endpoint in-flight ceilings, isolation policy. (moved from /pi/, 2026-10-19)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user