serverAppConfigs/serverAppConfigs-docs/Dankclaw/Voice Packs.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	How a voice pack is built: the files every pack must carry, the source-material gate, adding a voice, engine-bump drift.	dankclaw frank-tts voices tts	2026-10-18

Voice Packs

A voice pack is the unit of identity for frank-tts: one directory under /home/dank/voices, named by the voice, carrying the reference audio that makes the name real plus the pinned engine's own speaker artifacts and a provenance record. voice=narrator1 on POST /v1/audio/speech resolves to /home/dank/voices/narrator1/reference.wav and to nothing else — a name without that file is a 400 that lists the names that do exist, because a silent fallback voice is how a wrong-voice endpoint stays wrong for weeks.

Host: Dankclaw (192.168.0.143) — Pack root: /home/dank/voices, host state that stays out of the repo, mounted read-only at /voices by the façade unit — Builder: scripts/build-voice-pack.sh with tunables in scripts/voice-packs.env — Seam: tests/voice_pack_unit.bats — Engine: arm A qwentts.cpp, decided in .

Packs are world-readable on purpose (drwxr-xr-x, owner dank): the façade container runs as uid 10000 and reads the mount from outside its own uid map.

The required files

File	What it is	Who reads it
reference.wav	the crop: 24 kHz mono pcm_s16le, 18–25 s of clean single-speaker audio	the façade (it is the existence gate for the name) and the engine's boot warmer, which base64s it and registers the voice with no transcript
reference.spk	the engine's x-vector: 8,192 B = enc_dim 2048 × f32	the engine, for the ICL/--ref-spk path
reference.rvq	the engine's RVQ code stream: 5,456–6,864 B, 16 codebooks × T frames packed at 11 bits = 22 B per 12.5 Hz frame	the engine, for the ICL/--ref-rvq path
manifest.json	provenance: source path and sha256, crop start/end offsets, sample rate, duration, the tool and engine commit that made the artifacts, and the voice name	humans, and the builder itself on its second gate assertion

qwen-codec.log rides along with the extractor's own record of the run; the façade ignores anything it does not name and never deletes anything.

Division of enforcement, which is deliberate and not a gap: the façade gates on reference.wav because that is the only engine-agnostic term (see the voice-pack law in scripts/frank-tts.env); the builder is what refuses to emit a pack without its engine artifacts; the engine registers a voice from the WAV alone because its own POST /v1/audio/voices refuses a lone embedding — it wants wav_b64 or both spk_b64 and rvq_b64. So the artifacts are in the pack to make ICL a per-voice opt-in, not to boot it.

How a pack is built

# everything except the extractor: gate, crop, probe, plan, the manifest it would write
scripts/build-voice-pack.sh --dry-run narrator1 /home/dank/voices

# the real thing, on the host that holds the engine build
scripts/build-voice-pack.sh narrator1 /home/dank/voices

Crops come from scripts/voice-packs.env — per voice: source path, offset, duration, and the source's pinned sha256 — and any of them can be overridden on the command line (--source, --offset, --duration, --sha256, --tool). --all walks VOICE_PACKS. The builder assumes no GPU, no container, no fleet endpoint and no network: ffmpeg cuts the crop, the pinned extractor makes the artifacts, and both run nice -n 19 so a build never competes with the text endpoints on that box.

The extractor is qwen-codec in the qwentts.cpp tree at VOICE_PACK_ENGINE_COMMIT (5916bf7c79dbb4e66fc059cdc507a79a23a28bb4). Its usage text, read out of that tree and not guessed:

Usage: qwen-codec --model <gguf> [-i <input>] [--talker <gguf>] [--format <fmt>]

Required:
  --model <gguf>          Codec GGUF (qwen-tokenizer-12hz-*.gguf)

Optional:
  --device <name>         Compute device: CUDA0, Vulkan0, MTL0, CPU...
                          (default: best available)
  -i <path>               Input. WAV -> encode, .rvq -> decode
  --talker <gguf>         Talker GGUF (Base only). Encode also extracts the speaker
                          embedding and writes it next to the .rvq as a .spk file
  --format <fmt>          WAV output format: wav16, wav24, wav32 (default: wav16)

Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav.

Three consequences shape the builder. There is no -o, so the pack's WAV is staged under its final stem and the artifacts land beside it with nothing to rename. --talker must be a Base talker — the customvoice and voicedesign GGUFs carry no speaker encoder and the tool says has no speaker encoder (Base only). Encode truncates to the hop boundary, so the .rvq size is arithmetic, not chance: 22 × floor(samples / 1920) bytes, which is where the recorded 5,456–6,864 B window comes from and why the effective crop window is 19.84–25 s even though the WAV law says 18–25 s. A crop shorter than that is refused before the extractor runs, with the number in the message.

The build is staged and published, never assembled in place: reference.wav is the file the façade reads, so it moves into position last, after the artifacts and the manifest. A failed extractor therefore leaves a pack directory that is not a voice. Nothing is ever deleted — an existing pack is refused unless --force, and --force moves the whole old directory to /home/dank/voices/.AGENTTRASH/<voice>-<UTC>/ and builds fresh beside it.

Measured on the shipped narrator1 crop (2026-10-18, --dry-run against the real source): 98.739–118.636 s → 955,378 B of 24 kHz mono pcm_s16le, 19.897 s, planned .rvq 5,456 B — the same byte counts the trial produced by hand, and the extractor itself costs 1.8–2.1 s and 1.3–1.5 GiB peak per voice.

The 10-second gate

Project-wide source-material invariant: the first 10 seconds of any sample file used to generate voices is forbidden material. An offset that starts inside it is refused with exit 1 and a message that names the invariant, before a directory is created:

ERROR: source-material invariant violated: the crop must start past the first
10 s of the source file, and 'narrator1' asks for 4.5 s into … — nothing was
written.

The same gate is then asserted from the manifest that was just written — .crop.offset_seconds >= .invariant.forbidden_source_seconds, plus the recorded sha256 against the source it claims — so a pack cannot reach disk carrying a forbidden crop even if the offsets in the env file are edited into nonsense. 10 is not a knob: scripts/voice-packs.env carries the offsets, the sample rate, the windows and the engine facts, but the forbidden region is a constant in the builder. tests/voice_pack_unit.bats pins both halves (a refusal that creates nothing, and one legal crop that builds the whole pack) and applies the invariant to every crop in the tracked pack list.

Adding a fourth voice

1. Pick the source file and hash it: sha256sum /path/to/sample.wav. A crop offset is a claim about one exact file, so the digest goes in the env with it.

2. Find a speech-dense stretch past 10 s. The trial's recipe, applied once: ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null - build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the .rvq falls under the floor.

3. Add the four lines to scripts/voice-packs.env and the name to VOICE_PACKS:

   VOICE_PACKS="narrator1 narrator2 narrator3 narrator4"
   
   VOICE_PACK_NARRATOR4_SOURCE=/home/dank/voice-sources/audiosample4.m4b
   VOICE_PACK_NARRATOR4_OFFSET=142.31
   VOICE_PACK_NARRATOR4_DURATION=24
   VOICE_PACK_NARRATOR4_SHA256=<64 hex>
   

   The name must be lowercase [a-z0-9._-]: the façade matches the directory name verbatim while tts-server lowercases names internally, so an uppercase name would be reported one way by the façade and another by the engine. The builder refuses one anyway.

4. scripts/build-voice-pack.sh --dry-run narrator4 /home/dank/voices, read the plan, then drop --dry-run.

5. ./tests/run.sh (the pack-list invariant test covers the new row for free).

6. Register it with the running engine. The engine's voice registry is memory-only and warm-on-boot, but a restart re-decodes every voice (6–8 s each), so adding one pack uses the hook instead:

   podman exec frank-tts-engine /app/warm-voices.sh
   curl -s localhost:7772/v1/audio/voices      # the façade's list, not the engine's
   

   The hook walks /voices and registers every pack that has a reference.wav, on every run — the engine replaces a name it already holds (tools/tts-server.cpp:148 at the pinned commit: "Re-registering a name replaces the previous entry.") — then synthesizes twice per voice and logs voice <name> cold=…ms warm=…ms to the unit's journal. The two numbers should be close; see law 1 of Frank TTS Engine for what they measure. The rootless unit restart — systemctl --user restart frank-tts-engine.service — is for when the unit or its env changed, not for a new pack.

When the extractor's output size changes on an engine bump

The builder treats the two artifact sizes as properties of the pinned engine, so an engine bump that changes them fails the build: reference.spk is N B, expected 8192 B or reference.rvq is N B, expected …. That is the failure being loud, not a bug. Work through it in order:

1. Re-read the tool's own numbers rather than trusting this page: qwen-codec prints SpeakerEncoder] Loaded: enc_dim=… and Pipeline] Ready: hop … , … codebooks @ … Hz on every run, and the pinned tree's tools/qwen-codec.cpp documents the packing. enc_dim × 4 is the .spk codebooks × code_bits / 8 bytes per frame × frames is the .rvq.
2. Decide whether the new number is intended. A changed enc_dim means the old .spk files are unreadable by the new engine, so every pack must be rebuilt, not just the failing one.
3. Update the four arithmetic lines in scripts/voice-packs.env (VOICE_PACK_SPK_BYTES, VOICE_PACK_RVQ_MIN_BYTES, VOICE_PACK_RVQ_MAX_BYTES and VOICE_PACK_RVQ_BYTES_PER_FRAME if the codebook packing changed) and VOICE_PACK_ENGINE_COMMIT, VOICE_PACK_TOOL, VOICE_PACK_CODEC_MODEL VOICE_PACK_TALKER_MODEL to the new build. An env diff that moves the sizes without moving the commit is the mistake to avoid.
4. Rebuild all three packs with --force (the retired directories keep the old bytes readable beside their replacements until a human clears them), then re-run the warm hook: podman exec frank-tts-engine /app/warm-voices.sh. That hook registers every pack it finds on every run, and the engine replaces a name it already holds — tools/tts-server.cpp:148 at the pinned commit says so in its own words: "Re-registering a name replaces the previous entry." So the rebuilt reference.wav is what the running engine speaks afterwards, and the hook's cold/warm pass is what re-snapshots it. A restart (systemctl --user restart frank-tts-engine.service) is what an ENGINE bump needs — the registry is memory-only and lives in that process — and the unit runs the same hook on every start anyway, so a restart is never worse, only slower.
5. ./tests/voice_pack_unit.bats — the stub in that seam reproduces the packing law, so it needs the same arithmetic update as the env file. The seam's engine-bump test is exactly this scenario with a wrong .spk size, and its assertion — no reference.wav, no manifest — is the property that matters.

Host state (census 2026-10-18, read-only)

• /home/dank/voices/ holds narrator1, narrator2, narrator3, each with reference.wav, a reference16k.wav, and a manifest.json in the trial's own schema (kind: voice-reference-pack, a silence-scan record). They carry no reference.spk / reference.rvq: the trial's artifacts live in its own work directory (/home/dank/voice-audition/qwentts/voices/, and only for narrator1 and narrator2). The builder's first real run is therefore a migration, and it should be a --force run per voice so the old directories are retired rather than left half-updated.
• The three source files (/home/blank/audiosample.mp3 audiosample2.m4b, audiosample3.m4b, digests pinned in scripts/voice-packs.env) live on the workstation, not on Dankclaw; the builder reads them from $VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources, so they have to be copied there once before a build.
• VOICE_PACK_TOOL points at the trial's build tree (/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec, checked out at exactly 5916bf7c), and the two GGUFs live at /home/dank/models/Qwen3-TTS-GGUF/, the engine slice's fetch target — that path and its fetch state belong to Frank TTS Engine.
• Whether the façade (7772) and the engine (8880) are up and serving is deploy state on Frank TTS / Frank TTS Engine — read it there, not from this census.

Known gaps, on purpose

• No transcript in the manifest. ICL wants a ref_text that matches the crop, and the trial's machine transcripts came back wrong ("Demon Souls" for Dark Souls). Transcribing on the host is not a fix, so the pack carries none and the engine registers the WAV-only voice, which is what its boot warmer does anyway. When ICL becomes a per-voice opt-in, the curated transcript belongs in this manifest — a human-supplied field, not something the builder can derive.
• No 16 kHz variant. The packs the trial left behind carry reference16k.wav for the llama-tts-cli rollback engine; arm A is native 24 kHz, so the builder emits one reference. --force retires the whole directory including that variant, so if the rollback engine is ever revived: ffmpeg -i reference.wav -ar 16000 -c:a pcm_s16le reference16k.wav.
• No loudness pass. The trial's pipeline also measured and normalised (two-pass loudnorm, astats, ebur128, spectrograms) to a -16 LUFS spec. None of that is in the pack law, which asks for 24 kHz mono inside the duration window, so the builder does only that; a future engine that is loudness-sensitive should get the pass as an explicit, documented step.