dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into feat/frank-tts-engine

Browse Source 
# Conflicts:
#	serverAppConfigs-docs/index.md
#	serverAppConfigs-docs/log.md
This commit is contained in:
Frank
2026-10-18 17:44:31 -07:00
parent 8fefe6e1a9 d421cb69cf
commit 1312c29fe2
8 changed files with 1190 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Dankclaw/Frank TTS.md
+3 -2
View File
@@ -66,8 +66,9 @@ manifest.json source file, offsets, engine, sha256
```
`reference.wav` is the engine-agnostic term the façade gates on; the other two
are what makes a voice reusable by this engine, and the pack builder produces
them — never by hand, never fabricated from the WAV. Two laws apply to the
are what makes a voice reusable by this engine, and
[[Voice Packs|the pack builder]] produces them — never by hand, never
fabricated from the WAV. Two laws apply to the
source material: offsets always start **past the first 10 seconds** of the
sample, and packs are host state that stays out of this repo. Because the
container runs as uid 10000, files must be world-readable (`644`/`755`), and the
serverAppConfigs-docs/Dankclaw/Voice Packs.md
+251
View File
@@ -0,0 +1,251 @@
---
type: reference
description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes."
tags:
- dankclaw
- frank-tts
- voices
- tts
timestamp: 2026-10-18
---
# Voice Packs
A **voice pack** is the unit of identity for **frank-tts**: one directory under
`/home/dank/voices`, named by the voice, carrying the reference audio that makes
the name real plus the pinned engine's own speaker artifacts and a provenance
record. `voice=narrator1` on `POST /v1/audio/speech` resolves to
`/home/dank/voices/narrator1/reference.wav` and to nothing else — a name without
that file is a 400 that lists the names that do exist, because a silent fallback
voice is how a wrong-voice endpoint stays wrong for weeks.
**Host:** Dankclaw (192.168.0.143) — **Pack root:** `/home/dank/voices`, host
state that stays out of the repo, mounted read-only at `/voices` by the façade
unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in
`scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:**
arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]].
Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade
container runs as uid 10000 and reads the mount from outside its own uid map.
## The required files
| File | What it is | Who reads it |
|------|------------|--------------|
| `reference.wav` | the crop: 24 kHz mono `pcm_s16le`, **18–25 s** of clean single-speaker audio | the façade (it is the existence gate for the name) and the engine's boot warmer, which base64s it and registers the voice with no transcript |
| `reference.spk` | the engine's x-vector: **8,192 B** = `enc_dim` 2048 × f32 | the engine, for the ICL/`--ref-spk` path |
| `reference.rvq` | the engine's RVQ code stream: **5,456–6,864 B**, 16 codebooks × T frames packed at 11 bits = 22 B per 12.5 Hz frame | the engine, for the ICL/`--ref-rvq` path |
| `manifest.json` | provenance: source path and sha256, crop start/end offsets, sample rate, duration, the tool and engine commit that made the artifacts, and the `voice` name | humans, and the builder itself on its second gate assertion |
`qwen-codec.log` rides along with the extractor's own record of the run; the
façade ignores anything it does not name and never deletes anything.
Division of enforcement, which is deliberate and not a gap: the **façade** gates
on `reference.wav` because that is the only engine-agnostic term (see the
voice-pack law in `scripts/frank-tts.env`); the **builder** is what refuses to
emit a pack without its engine artifacts; the **engine** registers a voice from
the WAV alone because its own `POST /v1/audio/voices` refuses a lone embedding —
it wants `wav_b64` or **both** `spk_b64` and `rvq_b64`. So the artifacts are in
the pack to make ICL a per-voice opt-in, not to boot it.
## How a pack is built
```bash
# everything except the extractor: gate, crop, probe, plan, the manifest it would write
scripts/build-voice-pack.sh --dry-run narrator1 /home/dank/voices
# the real thing, on the host that holds the engine build
scripts/build-voice-pack.sh narrator1 /home/dank/voices
```
Crops come from `scripts/voice-packs.env` — per voice: source path, offset,
duration, and the source's pinned sha256 — and any of them can be overridden on
the command line (`--source`, `--offset`, `--duration`, `--sha256`, `--tool`).
`--all` walks `VOICE_PACKS`. The builder assumes no GPU, no container, no fleet
endpoint and no network: `ffmpeg` cuts the crop, the pinned extractor makes the
artifacts, and both run `nice -n 19` so a build never competes with the text
endpoints on that box.
The extractor is `qwen-codec` in the `qwentts.cpp` tree at
`VOICE_PACK_ENGINE_COMMIT` (`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`). Its
usage text, read out of that tree and not guessed:
```
Usage: qwen-codec --model <gguf> [-i <input>] [--talker <gguf>] [--format <fmt>]
Required:
--model <gguf> Codec GGUF (qwen-tokenizer-12hz-*.gguf)
Optional:
--device <name> Compute device: CUDA0, Vulkan0, MTL0, CPU...
(default: best available)
-i <path> Input. WAV -> encode, .rvq -> decode
--talker <gguf> Talker GGUF (Base only). Encode also extracts the speaker
embedding and writes it next to the .rvq as a .spk file
--format <fmt> WAV output format: wav16, wav24, wav32 (default: wav16)
Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav.
```
Three consequences shape the builder. **There is no `-o`**, so the pack's WAV is
staged under its final stem and the artifacts land beside it with nothing to
rename. **`--talker` must be a _Base_ talker** — the customvoice and voicedesign
GGUFs carry no speaker encoder and the tool says `has no speaker encoder (Base
only)`. **Encode truncates to the hop boundary**, so the `.rvq` size is arithmetic,
not chance: `22 × floor(samples / 1920)` bytes, which is where the recorded
5,456–6,864 B window comes from and why the **effective crop window is
19.84–25 s** even though the WAV law says 18–25 s. A crop shorter than that is
refused before the extractor runs, with the number in the message.
The build is staged and published, never assembled in place: `reference.wav` is
the file the façade reads, so it moves into position **last**, after the
artifacts and the manifest. A failed extractor therefore leaves a pack directory
that is not a voice. Nothing is ever deleted — an existing pack is refused unless
`--force`, and `--force` moves the whole old directory to
`/home/dank/voices/.AGENTTRASH/<voice>-<UTC>/` and builds fresh beside it.
Measured on the shipped `narrator1` crop (2026-10-18, `--dry-run` against the
real source): 98.739–118.636 s → `955,378 B` of 24 kHz mono `pcm_s16le`, 19.897
s, planned `.rvq` 5,456 B — the same byte counts the trial produced by hand, and
the extractor itself costs 1.8–2.1 s and 1.3–1.5 GiB peak per voice.
## The 10-second gate
Project-wide source-material invariant: **the first 10 seconds of any sample
file used to generate voices is forbidden material.** An offset that starts
inside it is refused with exit 1 and a message that names the invariant, before
a directory is created:
```
ERROR: source-material invariant violated: the crop must start past the first
10 s of the source file, and 'narrator1' asks for 4.5 s into … — nothing was
written.
```
The same gate is then asserted **from the manifest** that was just written —
`.crop.offset_seconds >= .invariant.forbidden_source_seconds`, plus the recorded
sha256 against the source it claims — so a pack cannot reach disk carrying a
forbidden crop even if the offsets in the env file are edited into nonsense.
`10` is not a knob: `scripts/voice-packs.env` carries the offsets, the sample
rate, the windows and the engine facts, but the forbidden region is a constant in
the builder. `tests/voice_pack_unit.bats` pins both halves (a refusal that
creates nothing, and one legal crop that builds the whole pack) and applies the
invariant to every crop in the tracked pack list.
## Adding a fourth voice
1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop
offset is a claim about one exact file, so the digest goes in the env with it.
2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once:
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`,
build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest
on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the
`.rvq` falls under the floor.
3. Add the four lines to `scripts/voice-packs.env` and the name to
`VOICE_PACKS`:
```
VOICE_PACKS="narrator1 narrator2 narrator3 narrator4"
VOICE_PACK_NARRATOR4_SOURCE=/home/dank/voice-sources/audiosample4.m4b
VOICE_PACK_NARRATOR4_OFFSET=142.31
VOICE_PACK_NARRATOR4_DURATION=24
VOICE_PACK_NARRATOR4_SHA256=<64 hex>
```
The name must be lowercase `[a-z0-9._-]`: the façade matches the directory
name verbatim while `tts-server` lowercases names internally, so an uppercase
name would be reported one way by the façade and another by the engine. The
builder refuses one anyway.
4. `scripts/build-voice-pack.sh --dry-run narrator4 /home/dank/voices`, read the
plan, then drop `--dry-run`.
5. `./tests/run.sh` (the pack-list invariant test covers the new row for free).
6. Register it with the running engine. The engine's voice registry is
memory-only and warm-on-boot, but a restart re-decodes **every** voice (6–8 s
each), so adding one pack uses the hook instead:
```bash
podman exec frank-tts-engine /app/warm-voices.sh
curl -s localhost:7772/v1/audio/voices # the façade's list, not the engine's
```
The hook walks `/voices`, registers any pack with a `reference.wav` it has not
seen, and logs `voice <name> cold=…ms warm=…ms` to the unit's journal. The
rootless unit restart — `systemctl --user restart frank-tts-engine.service` —
is for when the unit or its env changed, not for a new pack.
## When the extractor's output size changes on an engine bump
The builder treats the two artifact sizes as properties of the pinned engine, so
an engine bump that changes them **fails the build**: `reference.spk is N B,
expected 8192 B` or `reference.rvq is N B, expected …`. That is the failure being
loud, not a bug. Work through it in order:
1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec`
prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , …
codebooks @ … Hz` on every run, and the pinned tree's
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`;
`codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`.
2. Decide whether the new number is intended. A changed `enc_dim` means the old
`.spk` files are unreadable by the new engine, so **every** pack must be
rebuilt, not just the failing one.
3. Update the four arithmetic lines in `scripts/voice-packs.env`
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`,
and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and**
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`,
`VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes
without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old
bytes readable beside their replacements until a human clears them), then
restart the engine rather than using the warm hook: the hook only registers
packs it has **not** seen, and every voice here is already registered against
the old artifacts. `systemctl --user restart frank-tts-engine.service`
re-warms them all from the new bytes.
5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing
law, so it needs the same arithmetic update as the env file. The seam's
engine-bump test is exactly this scenario with a wrong `.spk` size, and its
assertion — no `reference.wav`, no manifest — is the property that matters.
## Host state (census 2026-10-18, read-only)
- `/home/dank/voices/` holds `narrator1`, `narrator2`, `narrator3`, each with
`reference.wav`, a `reference16k.wav`, and a `manifest.json` **in the trial's
own schema** (`kind: voice-reference-pack`, a silence-scan record). They carry
**no** `reference.spk` / `reference.rvq`: the trial's artifacts live in its own
work directory (`/home/dank/voice-audition/qwentts/voices/`, and only for
narrator1 and narrator2). The builder's first real run is therefore a
migration, and it should be a `--force` run per voice so the old directories
are retired rather than left half-updated.
- The three source files (`/home/blank/audiosample.mp3`,
`audiosample2.m4b`, `audiosample3.m4b`, digests pinned in
`scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the
builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so
they have to be copied there once before a build.
- `VOICE_PACK_TOOL` points at the trial's build tree
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
at exactly `5916bf7c`), and the two GGUFs at
`/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's
download script puts them — that directory does not exist yet, so until it
does the models have to come from the trial's `models/` directory.
- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs
is possible on that host while serving them is not.
## Known gaps, on purpose
- **No transcript in the manifest.** ICL wants a `ref_text` that matches the
crop, and the trial's machine transcripts came back wrong ("Demon Souls" for
*Dark Souls*). Transcribing on the host is not a fix, so the pack carries none
and the engine registers the WAV-only voice, which is what its boot warmer
does anyway. When ICL becomes a per-voice opt-in, the curated transcript
belongs in this manifest — a human-supplied field, not something the builder
can derive.
- **No 16 kHz variant.** The packs the trial left behind carry
`reference16k.wav` for the `llama-tts-cli` rollback engine; arm A is native
24 kHz, so the builder emits one reference. `--force` retires the whole
directory including that variant, so if the rollback engine is ever revived:
`ffmpeg -i reference.wav -ar 16000 -c:a pcm_s16le reference16k.wav`.
- **No loudness pass.** The trial's pipeline also measured and normalised
(two-pass `loudnorm`, `astats`, `ebur128`, spectrograms) to a `-16 LUFS` spec.
None of that is in the pack law, which asks for 24 kHz mono inside the
duration window, so the builder does only that; a future engine that is
loudness-sensitive should get the pass as an explicit, documented step.
serverAppConfigs-docs/index.md
+1
View File
@@ -82,6 +82,7 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; repo-owned config, not deployed
- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; repo-owned config, not deployed
- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
## Infrastructure
serverAppConfigs-docs/log.md
+53
View File
@@ -2536,3 +2536,56 @@ every restart since the cutover.
comparison, planner separation and order, the façade handler's steps, the
homepage card's gating, and routing that selects both seams for either
deployable's paths.
## 2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code
- Third slice of the voice stack (after the façade and the engine decision):
`scripts/build-voice-pack.sh` turns a source recording plus a crop offset into
a deployable pack — `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the
engine's own `reference.spk` (8,192 B) + `reference.rvq` (5,456–6,864 B) made
from it, and `manifest.json` provenance. Tracked tunables in
`scripts/voice-packs.env`: pack list, per-voice source/offset/duration/pinned
source sha256, extractor path and engine commit, artifact arithmetic. New runbook
[[Voice Packs]].
- **The gate is the product.** The project-wide invariant — no voice material
from the first 10 s of a sample file — had only ever been a sentence in
`scripts/frank-tts.env`. An offset inside it now exits 1 before a directory is
created, and the same assertion runs again against the manifest that was just
written, next to the recorded sha256, so a pack cannot carry a forbidden crop
even if the env file is edited into nonsense. `10` stays a constant in the
builder; the env file carries offsets, never the law.
- **The extractor's contract was read, not guessed:** `qwen-codec` in qwentts.cpp
`5916bf7c` takes `--model <codec gguf> --talker <base talker gguf> -i <wav>`
with an optional `--device`, has **no `-o`** and names both artifacts next to
the input with the extension swapped. So the WAV is staged under its final stem
and `reference.wav` — the file the façade gates on — is moved into place
**last**, after the artifacts and the manifest: a failed extractor leaves a
directory that is not a voice. `--force` retires an existing pack whole to
`.AGENTTRASH/` instead of touching bytes; nothing in this script deletes.
- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B per
12.5 Hz frame (`22 × floor(samples/1920)`) and `enc_dim 2048 × f32` = 8,192 B.
That makes the **effective crop window 19.84–25 s**, narrower than the WAV
law's 18–25 s, and a too-short crop is refused before the extractor runs. An
engine bump that changes either number fails the build by design; the bump
procedure is in [[Voice Packs]].
- Verified against reality, not against the model: the pinned `narrator1` crop
(98.739–118.636 s of `audiosample.mp3`, sha256 `2299510…`) comes out of a
`--dry-run` at 955,378 B / 19.897 s with 5,456 B of `.rvq` planned — byte-for-byte
the trial's hand-cut numbers — and all three pinned source digests match the
files on disk.
- Seam: `tests/voice_pack_unit.bats` (13 tests) — refusal inside the forbidden
region creates nothing, a legal crop builds the whole pack, `ffprobe` on the
generated reference, manifest round-trip through `jq` with a sha that matches
the source, re-run-without-`--force` leaves every byte alone, `--force` retires
rather than deletes, `--dry-run` needs no extractor, a wrong-sized artifact
from a bumped engine cannot publish a voice, and every crop in the tracked pack
list clears the invariant. `qwen-codec` is stood in by a stub with the real
tool's I/O shape, so no engine, no GPU, no network, no fleet host. Suite green:
**416/416** (`./tests/run.sh`, wall 77 s) — 403 before this branch.
- Deliberately NOT done: no transcript in the manifest (ICL wants a curated
`ref_text`; the trial's machine transcripts were wrong, and the engine
registers WAV-only voices anyway), no 16 kHz variant (arm A is native 24 kHz),
no loudness pass (not in the pack law), and no pack built on the host — the
three sources still live on the workstation and the engine GGUFs have not been
fetched to `/home/dank/models/Qwen3-TTS-GGUF` yet, so the first real run is a
host step.