diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md new file mode 100644 index 0000000..ce23c2d --- /dev/null +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -0,0 +1,251 @@ +--- +type: reference +description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes." +tags: + - dankclaw + - frank-tts + - voices + - tts +timestamp: 2026-10-18 +--- + +# Voice Packs + +A **voice pack** is the unit of identity for **frank-tts**: one directory under +`/home/dank/voices`, named by the voice, carrying the reference audio that makes +the name real plus the pinned engine's own speaker artifacts and a provenance +record. `voice=narrator1` on `POST /v1/audio/speech` resolves to +`/home/dank/voices/narrator1/reference.wav` and to nothing else — a name without +that file is a 400 that lists the names that do exist, because a silent fallback +voice is how a wrong-voice endpoint stays wrong for weeks. + +**Host:** Dankclaw (192.168.0.143) — **Pack root:** `/home/dank/voices`, host +state that stays out of the repo, mounted read-only at `/voices` by the façade +unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in +`scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:** +arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]]. + +Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade +container runs as uid 10000 and reads the mount from outside its own uid map. + +## The required files + +| File | What it is | Who reads it | +|------|------------|--------------| +| `reference.wav` | the crop: 24 kHz mono `pcm_s16le`, **18–25 s** of clean single-speaker audio | the façade (it is the existence gate for the name) and the engine's boot warmer, which base64s it and registers the voice with no transcript | +| `reference.spk` | the engine's x-vector: **8,192 B** = `enc_dim` 2048 × f32 | the engine, for the ICL/`--ref-spk` path | +| `reference.rvq` | the engine's RVQ code stream: **5,456–6,864 B**, 16 codebooks × T frames packed at 11 bits = 22 B per 12.5 Hz frame | the engine, for the ICL/`--ref-rvq` path | +| `manifest.json` | provenance: source path and sha256, crop start/end offsets, sample rate, duration, the tool and engine commit that made the artifacts, and the `voice` name | humans, and the builder itself on its second gate assertion | + +`qwen-codec.log` rides along with the extractor's own record of the run; the +façade ignores anything it does not name and never deletes anything. + +Division of enforcement, which is deliberate and not a gap: the **façade** gates +on `reference.wav` because that is the only engine-agnostic term (see the +voice-pack law in `scripts/frank-tts.env`); the **builder** is what refuses to +emit a pack without its engine artifacts; the **engine** registers a voice from +the WAV alone because its own `POST /v1/audio/voices` refuses a lone embedding — +it wants `wav_b64` or **both** `spk_b64` and `rvq_b64`. So the artifacts are in +the pack to make ICL a per-voice opt-in, not to boot it. + +## How a pack is built + +```bash +# everything except the extractor: gate, crop, probe, plan, the manifest it would write +scripts/build-voice-pack.sh --dry-run narrator1 /home/dank/voices + +# the real thing, on the host that holds the engine build +scripts/build-voice-pack.sh narrator1 /home/dank/voices +``` + +Crops come from `scripts/voice-packs.env` — per voice: source path, offset, +duration, and the source's pinned sha256 — and any of them can be overridden on +the command line (`--source`, `--offset`, `--duration`, `--sha256`, `--tool`). +`--all` walks `VOICE_PACKS`. The builder assumes no GPU, no container, no fleet +endpoint and no network: `ffmpeg` cuts the crop, the pinned extractor makes the +artifacts, and both run `nice -n 19` so a build never competes with the text +endpoints on that box. + +The extractor is `qwen-codec` in the `qwentts.cpp` tree at +`VOICE_PACK_ENGINE_COMMIT` (`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`). Its +usage text, read out of that tree and not guessed: + +``` +Usage: qwen-codec --model [-i ] [--talker ] [--format ] + +Required: + --model Codec GGUF (qwen-tokenizer-12hz-*.gguf) + +Optional: + --device Compute device: CUDA0, Vulkan0, MTL0, CPU... + (default: best available) + -i Input. WAV -> encode, .rvq -> decode + --talker Talker GGUF (Base only). Encode also extracts the speaker + embedding and writes it next to the .rvq as a .spk file + --format WAV output format: wav16, wav24, wav32 (default: wav16) + +Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav. +``` + +Three consequences shape the builder. **There is no `-o`**, so the pack's WAV is +staged under its final stem and the artifacts land beside it with nothing to +rename. **`--talker` must be a _Base_ talker** — the customvoice and voicedesign +GGUFs carry no speaker encoder and the tool says `has no speaker encoder (Base +only)`. **Encode truncates to the hop boundary**, so the `.rvq` size is arithmetic, +not chance: `22 × floor(samples / 1920)` bytes, which is where the recorded +5,456–6,864 B window comes from and why the **effective crop window is +19.84–25 s** even though the WAV law says 18–25 s. A crop shorter than that is +refused before the extractor runs, with the number in the message. + +The build is staged and published, never assembled in place: `reference.wav` is +the file the façade reads, so it moves into position **last**, after the +artifacts and the manifest. A failed extractor therefore leaves a pack directory +that is not a voice. Nothing is ever deleted — an existing pack is refused unless +`--force`, and `--force` moves the whole old directory to +`/home/dank/voices/.AGENTTRASH/-/` and builds fresh beside it. + +Measured on the shipped `narrator1` crop (2026-10-18, `--dry-run` against the +real source): 98.739–118.636 s → `955,378 B` of 24 kHz mono `pcm_s16le`, 19.897 +s, planned `.rvq` 5,456 B — the same byte counts the trial produced by hand, and +the extractor itself costs 1.8–2.1 s and 1.3–1.5 GiB peak per voice. + +## The 10-second gate + +Project-wide source-material invariant: **the first 10 seconds of any sample +file used to generate voices is forbidden material.** An offset that starts +inside it is refused with exit 1 and a message that names the invariant, before +a directory is created: + +``` +ERROR: source-material invariant violated: the crop must start past the first +10 s of the source file, and 'narrator1' asks for 4.5 s into … — nothing was +written. +``` + +The same gate is then asserted **from the manifest** that was just written — +`.crop.offset_seconds >= .invariant.forbidden_source_seconds`, plus the recorded +sha256 against the source it claims — so a pack cannot reach disk carrying a +forbidden crop even if the offsets in the env file are edited into nonsense. +`10` is not a knob: `scripts/voice-packs.env` carries the offsets, the sample +rate, the windows and the engine facts, but the forbidden region is a constant in +the builder. `tests/voice_pack_unit.bats` pins both halves (a refusal that +creates nothing, and one legal crop that builds the whole pack) and applies the +invariant to every crop in the tracked pack list. + +## Adding a fourth voice + +1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop + offset is a claim about one exact file, so the digest goes in the env with it. +2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once: + `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`, + build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest + on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the + `.rvq` falls under the floor. +3. Add the four lines to `scripts/voice-packs.env` and the name to + `VOICE_PACKS`: + + ``` + VOICE_PACKS="narrator1 narrator2 narrator3 narrator4" + + VOICE_PACK_NARRATOR4_SOURCE=/home/dank/voice-sources/audiosample4.m4b + VOICE_PACK_NARRATOR4_OFFSET=142.31 + VOICE_PACK_NARRATOR4_DURATION=24 + VOICE_PACK_NARRATOR4_SHA256=<64 hex> + ``` + + The name must be lowercase `[a-z0-9._-]`: the façade matches the directory + name verbatim while `tts-server` lowercases names internally, so an uppercase + name would be reported one way by the façade and another by the engine. The + builder refuses one anyway. +4. `scripts/build-voice-pack.sh --dry-run narrator4 /home/dank/voices`, read the + plan, then drop `--dry-run`. +5. `./tests/run.sh` (the pack-list invariant test covers the new row for free). +6. Register it with the running engine. The engine's voice registry is + memory-only and warm-on-boot, but a restart re-decodes **every** voice (6–8 s + each), so adding one pack uses the hook instead: + + ```bash + podman exec frank-tts-engine /app/warm-voices.sh + curl -s localhost:7772/v1/audio/voices # the façade's list, not the engine's + ``` + + The hook walks `/voices`, registers any pack with a `reference.wav` it has not + seen, and logs `voice cold=…ms warm=…ms` to the unit's journal. The + rootless unit restart — `systemctl --user restart frank-tts-engine.service` — + is for when the unit or its env changed, not for a new pack. + +## When the extractor's output size changes on an engine bump + +The builder treats the two artifact sizes as properties of the pinned engine, so +an engine bump that changes them **fails the build**: `reference.spk is N B, +expected 8192 B` or `reference.rvq is N B, expected …`. That is the failure being +loud, not a bug. Work through it in order: + +1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec` + prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , … + codebooks @ … Hz` on every run, and the pinned tree's + `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`; + `codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`. +2. Decide whether the new number is intended. A changed `enc_dim` means the old + `.spk` files are unreadable by the new engine, so **every** pack must be + rebuilt, not just the failing one. +3. Update the four arithmetic lines in `scripts/voice-packs.env` + (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`, + and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and** + `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`, + `VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes + without moving the commit is the mistake to avoid. +4. Rebuild all three packs with `--force` (the retired directories keep the old + bytes readable beside their replacements until a human clears them), then + restart the engine rather than using the warm hook: the hook only registers + packs it has **not** seen, and every voice here is already registered against + the old artifacts. `systemctl --user restart frank-tts-engine.service` + re-warms them all from the new bytes. +5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing + law, so it needs the same arithmetic update as the env file. The seam's + engine-bump test is exactly this scenario with a wrong `.spk` size, and its + assertion — no `reference.wav`, no manifest — is the property that matters. + +## Host state (census 2026-10-18, read-only) + +- `/home/dank/voices/` holds `narrator1`, `narrator2`, `narrator3`, each with + `reference.wav`, a `reference16k.wav`, and a `manifest.json` **in the trial's + own schema** (`kind: voice-reference-pack`, a silence-scan record). They carry + **no** `reference.spk` / `reference.rvq`: the trial's artifacts live in its own + work directory (`/home/dank/voice-audition/qwentts/voices/`, and only for + narrator1 and narrator2). The builder's first real run is therefore a + migration, and it should be a `--force` run per voice so the old directories + are retired rather than left half-updated. +- The three source files (`/home/blank/audiosample.mp3`, + `audiosample2.m4b`, `audiosample3.m4b`, digests pinned in + `scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the + builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so + they have to be copied there once before a build. +- `VOICE_PACK_TOOL` points at the trial's build tree + (`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out + at exactly `5916bf7c`), and the two GGUFs at + `/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's + download script puts them — that directory does not exist yet, so until it + does the models have to come from the trial's `models/` directory. +- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs + is possible on that host while serving them is not. + +## Known gaps, on purpose + +- **No transcript in the manifest.** ICL wants a `ref_text` that matches the + crop, and the trial's machine transcripts came back wrong ("Demon Souls" for + *Dark Souls*). Transcribing on the host is not a fix, so the pack carries none + and the engine registers the WAV-only voice, which is what its boot warmer + does anyway. When ICL becomes a per-voice opt-in, the curated transcript + belongs in this manifest — a human-supplied field, not something the builder + can derive. +- **No 16 kHz variant.** The packs the trial left behind carry + `reference16k.wav` for the `llama-tts-cli` rollback engine; arm A is native + 24 kHz, so the builder emits one reference. `--force` retires the whole + directory including that variant, so if the rollback engine is ever revived: + `ffmpeg -i reference.wav -ar 16000 -c:a pcm_s16le reference16k.wav`. +- **No loudness pass.** The trial's pipeline also measured and normalised + (two-pass `loudnorm`, `astats`, `ebur128`, spectrograms) to a `-16 LUFS` spec. + None of that is in the pack law, which asks for 24 kHz mono inside the + duration window, so the builder does only that; a future engine that is + loudness-sensitive should get the pass as an explicit, documented step. diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index fe38343..65571ad 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -80,6 +80,7 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver. - [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack) - [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress) - [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed +- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate ## Infrastructure diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 81b1454..74fba84 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2441,3 +2441,56 @@ every restart since the cutover. `~/.harbor/compose.opennotebook.yml` pins the deprecated `-single` image, carries no encryption key and injects the whole harbor env — left untouched, read-only, as the record of what not to copy. + +## 2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code + +- Third slice of the voice stack (after the façade and the engine decision): + `scripts/build-voice-pack.sh` turns a source recording plus a crop offset into + a deployable pack — `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the + engine's own `reference.spk` (8,192 B) + `reference.rvq` (5,456–6,864 B) made + from it, and `manifest.json` provenance. Tracked tunables in + `scripts/voice-packs.env`: pack list, per-voice source/offset/duration/pinned + source sha256, extractor path and engine commit, artifact arithmetic. New runbook + [[Voice Packs]]. +- **The gate is the product.** The project-wide invariant — no voice material + from the first 10 s of a sample file — had only ever been a sentence in + `scripts/frank-tts.env`. An offset inside it now exits 1 before a directory is + created, and the same assertion runs again against the manifest that was just + written, next to the recorded sha256, so a pack cannot carry a forbidden crop + even if the env file is edited into nonsense. `10` stays a constant in the + builder; the env file carries offsets, never the law. +- **The extractor's contract was read, not guessed:** `qwen-codec` in qwentts.cpp + `5916bf7c` takes `--model --talker -i ` + with an optional `--device`, has **no `-o`** and names both artifacts next to + the input with the extension swapped. So the WAV is staged under its final stem + and `reference.wav` — the file the façade gates on — is moved into place + **last**, after the artifacts and the manifest: a failed extractor leaves a + directory that is not a voice. `--force` retires an existing pack whole to + `.AGENTTRASH/` instead of touching bytes; nothing in this script deletes. +- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B per + 12.5 Hz frame (`22 × floor(samples/1920)`) and `enc_dim 2048 × f32` = 8,192 B. + That makes the **effective crop window 19.84–25 s**, narrower than the WAV + law's 18–25 s, and a too-short crop is refused before the extractor runs. An + engine bump that changes either number fails the build by design; the bump + procedure is in [[Voice Packs]]. +- Verified against reality, not against the model: the pinned `narrator1` crop + (98.739–118.636 s of `audiosample.mp3`, sha256 `2299510…`) comes out of a + `--dry-run` at 955,378 B / 19.897 s with 5,456 B of `.rvq` planned — byte-for-byte + the trial's hand-cut numbers — and all three pinned source digests match the + files on disk. +- Seam: `tests/voice_pack_unit.bats` (13 tests) — refusal inside the forbidden + region creates nothing, a legal crop builds the whole pack, `ffprobe` on the + generated reference, manifest round-trip through `jq` with a sha that matches + the source, re-run-without-`--force` leaves every byte alone, `--force` retires + rather than deletes, `--dry-run` needs no extractor, a wrong-sized artifact + from a bumped engine cannot publish a voice, and every crop in the tracked pack + list clears the invariant. `qwen-codec` is stood in by a stub with the real + tool's I/O shape, so no engine, no GPU, no network, no fleet host. Suite green: + **416/416** (`./tests/run.sh`, wall 77 s) — 403 before this branch. +- Deliberately NOT done: no transcript in the manifest (ICL wants a curated + `ref_text`; the trial's machine transcripts were wrong, and the engine + registers WAV-only voices anyway), no 16 kHz variant (arm A is native 24 kHz), + no loudness pass (not in the pack law), and no pack built on the host — the + three sources still live on the workstation and the engine GGUFs have not been + fetched to `/home/dank/models/Qwen3-TTS-GGUF` yet, so the first real run is a + host step.