Frank
2242fcfec0
Turns a source recording plus a crop offset into the pack frank-tts resolves by name: reference.wav (24 kHz mono pcm_s16le, 18-25 s), the engine's own reference.spk / reference.rvq made from it, and manifest.json provenance (source path + sha256, crop start/end, sample rate, duration, tool, engine commit, voice name). The extractor is qwen-codec from the pinned qwentts.cpp tree, and the flags are its own usage text, not a guess: --model <codec gguf>, --talker <base talker gguf>, -i <wav>, optional --device; there is no -o, so it names both artifacts next to the input with the extension swapped, which is why the pack's WAV is staged under its final stem. The 10-second source-material invariant is enforced, not annotated: an offset inside the forbidden region exits 1 before a directory is created, and the gate is asserted a second time from the manifest that was just written, together with the recorded sha256. Artifact sizes are checked against the codec's own arithmetic (8,192 B x-vector; 22 B per 12.5 Hz frame), so an engine bump fails the build instead of shipping a pack the engine rejects at synthesis time — and the crop is refused up front when it is too short to fill the .rvq window. reference.wav is the file the façade gates on, so it lands last, after the artifacts and the manifest: a voice name never exists on disk without its artifacts behind it. Nothing is ever deleted — an existing pack is refused unless --force, which retires it whole to .AGENTTRASH/ — and --dry-run runs every step except the extractor, so the seam can test the gate without the engine.
484 lines
24 KiB
Bash
Executable File
484 lines
24 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# build-voice-pack.sh — turn a SOURCE recording plus a CROP OFFSET into a
|
|
# deployable frank-tts voice pack: the reference WAV, the engine's own speaker
|
|
# artifacts made from it, and manifest.json provenance.
|
|
#
|
|
# THE PACK. $PACKS_DIR/<voice>/ carries the three members the pack law in
|
|
# scripts/frank-tts.env names:
|
|
#
|
|
# reference.wav the crop: 24 kHz mono PCM (pcm_s16le), 18-25 s of clean
|
|
# single-speaker audio. This is what makes a voice NAME real
|
|
# (dankclaw/voiceservice/app.py lists and resolves exactly
|
|
# this file), so it is written LAST, once everything else in
|
|
# the pack has been verified.
|
|
# reference.spk the engine's x-vector: enc_dim 2048 f32 = 8,192 B.
|
|
# reference.rvq the engine's RVQ code stream: 16 codebooks x 11 bits = 22 B
|
|
# per 12.5 Hz frame = 22 * floor(samples / 1920).
|
|
# manifest.json provenance: source path and sha256, crop start/end, sample
|
|
# rate, duration, the tool + engine commit that made the
|
|
# artifacts, and the voice name.
|
|
#
|
|
# Both artifacts come from the pinned engine build's speaker extractor — the
|
|
# `qwen-codec` tool of qwentts.cpp at VOICE_PACK_ENGINE_COMMIT — run on the pack's
|
|
# own WAV, and the extractor names them by swapping the input's extension, so
|
|
# they land beside reference.wav with nothing to rename. Required, never a
|
|
# replacement for the WAV: a voice another runtime cannot reuse is not saved.
|
|
#
|
|
# THE 10-SECOND GATE — THE REASON THIS SCRIPT EXISTS. Project-wide source-material
|
|
# invariant: the first 10 s of any sample file used to generate voices is
|
|
# forbidden material. An offset that starts inside that region is refused with a
|
|
# nonzero exit BEFORE anything is created — no pack dir, no WAV, no artifact — and
|
|
# the same gate is asserted a second time from the manifest that was just written
|
|
# (plus the recorded sha256 against the source it claims). A pack carrying a
|
|
# forbidden crop therefore cannot survive a build.
|
|
#
|
|
# THE ARTIFACT SIZES ARE CHECKED, NOT ASSUMED. They are a property of the codec
|
|
# config inside the GGUF, so an engine bump that changes enc_dim or the codebook
|
|
# count fails the build instead of shipping a pack the engine will reject at
|
|
# synthesis time. Note the interaction with the WAV window: an 18.0 s crop is a
|
|
# legal reference.wav but only 22 * 225 = 4,950 B of .rvq, under the recorded
|
|
# 5,456 B floor, so the effective crop window is 19.84-25 s. That is a real
|
|
# constraint and the builder says so up front rather than failing after a build;
|
|
# serverAppConfigs-docs/Dankclaw/Voice Packs.md covers what to do on an engine bump.
|
|
#
|
|
# WHERE IT RUNS: the host that holds the engine build (Dankclaw). The extractor is
|
|
# a CPU ggml binary — no GPU, no container, no fleet endpoint, no network. ffmpeg
|
|
# does the crop; every spawned process runs `nice -n 19` so a pack build never
|
|
# competes with the text endpoints on that box.
|
|
#
|
|
# IT NEVER DELETES. An existing pack is refused unless --force, and --force moves
|
|
# the whole old directory aside into $PACKS_DIR/.AGENTTRASH/<voice>-<UTC>/ rather
|
|
# than overwriting or unlinking a single byte.
|
|
#
|
|
# Usage:
|
|
# scripts/build-voice-pack.sh [OPTIONS] VOICE PACKS_DIR
|
|
# scripts/build-voice-pack.sh [OPTIONS] --all PACKS_DIR
|
|
#
|
|
# Options (anything not given comes from scripts/voice-packs.env):
|
|
# --env FILE tunables file (default scripts/voice-packs.env)
|
|
# --source FILE source recording for this voice
|
|
# --offset SEC crop start, in seconds into the source (must be >= 10)
|
|
# --duration SEC crop length; reference.wav comes out this long
|
|
# --sha256 HEX expected sha256 of the source file
|
|
# --tool PATH the pinned speaker extractor (qwen-codec)
|
|
# --all build every voice in VOICE_PACKS, skipping none
|
|
# --dry-run everything except the extractor: validate, cut the crop,
|
|
# probe it, print the plan and the manifest it would write
|
|
# --force retire an existing pack to .AGENTTRASH/ and rebuild
|
|
# -h, --help this text
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
ENV_FILE="$REPO_ROOT/scripts/voice-packs.env"
|
|
SOURCE_FORBIDDEN_SECONDS=10 # the source-material invariant; not a knob
|
|
WORK_DIR="" # scratch dir a dry run owns, removed on exit
|
|
|
|
# Only ever removes the scratch directory this script created for itself; pack
|
|
# bytes on disk are retired to .AGENTTRASH/, never deleted (see --force).
|
|
cleanup_work_dir() {
|
|
if [ -n "$WORK_DIR" ]; then
|
|
rm -rf -- "$WORK_DIR"
|
|
fi
|
|
}
|
|
trap cleanup_work_dir EXIT
|
|
|
|
die() {
|
|
printf 'ERROR: %s\n' "$*" >&2
|
|
exit 1
|
|
}
|
|
|
|
usage() {
|
|
sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'
|
|
}
|
|
|
|
# fcmp A OP B — float comparison (awk); OP is one of < <= > >= == and A/B are
|
|
# already validated as plain decimals, so nothing user-shaped reaches awk as
|
|
# program text.
|
|
fcmp() {
|
|
awk -v a="$1" -v b="$3" "BEGIN { exit !(a $2 b) }"
|
|
}
|
|
|
|
# floorn X — greatest integer <= X, printed. Only ever called on positives.
|
|
floorn() {
|
|
awk -v x="$1" 'BEGIN { printf "%d\n", int(x) }'
|
|
}
|
|
|
|
is_seconds() {
|
|
[[ "$1" =~ ^[0-9]+([.][0-9]+)?$ ]]
|
|
}
|
|
|
|
is_hex64() {
|
|
[[ "$1" =~ ^[0-9a-f]{64}$ ]]
|
|
}
|
|
|
|
# ---------------------------------------------------------------- arguments --
|
|
VOICE=""
|
|
PACKS_DIR=""
|
|
ARG_SOURCE=""
|
|
ARG_OFFSET=""
|
|
ARG_DURATION=""
|
|
ARG_SHA=""
|
|
ARG_TOOL=""
|
|
DRY_RUN=0
|
|
FORCE=0
|
|
BUILD_ALL=0
|
|
|
|
while [ "$#" -gt 0 ]; do
|
|
case "$1" in
|
|
--env) ENV_FILE="${2:?--env needs a path}"; shift 2 ;;
|
|
--source) ARG_SOURCE="${2:?--source needs a file}"; shift 2 ;;
|
|
--offset) ARG_OFFSET="${2:?--offset needs seconds}"; shift 2 ;;
|
|
--duration) ARG_DURATION="${2:?--duration needs seconds}"; shift 2 ;;
|
|
--sha256) ARG_SHA="${2:?--sha256 needs a hex digest}"; shift 2 ;;
|
|
--tool) ARG_TOOL="${2:?--tool needs a path}"; shift 2 ;;
|
|
--dry-run) DRY_RUN=1; shift ;;
|
|
--force) FORCE=1; shift ;;
|
|
--all) BUILD_ALL=1; shift ;;
|
|
-h|--help) usage; exit 0 ;;
|
|
--*) die "unknown option: $1 (see --help)" ;;
|
|
*)
|
|
if [ -z "$VOICE" ]; then
|
|
VOICE="$1"
|
|
elif [ -z "$PACKS_DIR" ]; then
|
|
PACKS_DIR="$1"
|
|
else
|
|
die "unexpected argument: $1"
|
|
fi
|
|
shift
|
|
;;
|
|
esac
|
|
done
|
|
|
|
[ -f "$ENV_FILE" ] || die "tunables file not found: $ENV_FILE"
|
|
for tool in ffmpeg ffprobe jq sha256sum; do
|
|
command -v "$tool" >/dev/null 2>&1 || die "$tool is required and not on PATH"
|
|
done
|
|
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
|
|
# Tunables the env file owes us. Defaults are the engine facts read out of the
|
|
# pinned tree; an env file that omits one still builds against the truth.
|
|
VOICE_PACKS="${VOICE_PACKS:-}"
|
|
SAMPLE_RATE="${VOICE_PACK_SAMPLE_RATE:-24000}"
|
|
MIN_SECONDS="${VOICE_PACK_MIN_SECONDS:-18}"
|
|
MAX_SECONDS="${VOICE_PACK_MAX_SECONDS:-25}"
|
|
FRAME_RATE="${VOICE_PACK_FRAME_RATE:-12.5}"
|
|
RVQ_BYTES_PER_FRAME="${VOICE_PACK_RVQ_BYTES_PER_FRAME:-22}"
|
|
SPK_BYTES="${VOICE_PACK_SPK_BYTES:-8192}"
|
|
RVQ_MIN_BYTES="${VOICE_PACK_RVQ_MIN_BYTES:-5456}"
|
|
RVQ_MAX_BYTES="${VOICE_PACK_RVQ_MAX_BYTES:-6864}"
|
|
TOOL="${ARG_TOOL:-${VOICE_PACK_TOOL:-}}"
|
|
CODEC_MODEL="${VOICE_PACK_CODEC_MODEL:-}"
|
|
TALKER_MODEL="${VOICE_PACK_TALKER_MODEL:-}"
|
|
ENGINE_REPO="${VOICE_PACK_ENGINE_REPO:-https://github.com/ServeurpersoCom/qwentts.cpp}"
|
|
ENGINE_COMMIT="${VOICE_PACK_ENGINE_COMMIT:-5916bf7c79dbb4e66fc059cdc507a79a23a28bb4}"
|
|
DEVICE="${VOICE_PACK_DEVICE:-CPU}"
|
|
THREADS="${VOICE_PACK_THREADS:-8}"
|
|
NICE_LEVEL="${VOICE_PACK_NICE:-19}"
|
|
TRASH_DIRNAME="${VOICE_PACK_TRASH_DIRNAME:-.AGENTTRASH}"
|
|
DEFAULT_PACKS_DIR="${VOICE_PACK_DIR:-/home/dank/voices}"
|
|
|
|
[ -n "$PACKS_DIR" ] || PACKS_DIR="$DEFAULT_PACKS_DIR"
|
|
[ -n "$PACKS_DIR" ] || die "no packs directory (pass it as an argument)"
|
|
is_seconds "$SAMPLE_RATE" || die "VOICE_PACK_SAMPLE_RATE is not a number: $SAMPLE_RATE"
|
|
is_seconds "$MIN_SECONDS" || die "VOICE_PACK_MIN_SECONDS is not a number: $MIN_SECONDS"
|
|
is_seconds "$MAX_SECONDS" || die "VOICE_PACK_MAX_SECONDS is not a number: $MAX_SECONDS"
|
|
|
|
# hop / frame arithmetic, derived once: the codec runs at FRAME_RATE Hz over
|
|
# SAMPLE_RATE audio, so one frame is HOP_SAMPLES samples and one second is
|
|
# FRAME_RATE frames.
|
|
HOP_SAMPLES="$(floorn "$(awk -v r="$SAMPLE_RATE" -v f="$FRAME_RATE" 'BEGIN { print r / f }')")"
|
|
[ "$HOP_SAMPLES" -gt 0 ] || die "VOICE_PACK_FRAME_RATE ($FRAME_RATE) exceeds the sample rate"
|
|
BYTES_PER_SECOND="$(awk -v b="$RVQ_BYTES_PER_FRAME" -v f="$FRAME_RATE" 'BEGIN { print b * f }')"
|
|
|
|
# rvq_bytes_for DURATION — packed size the extractor will emit for a crop this
|
|
# long: whole frames only, because it truncates to the hop boundary.
|
|
rvq_bytes_for() {
|
|
local samples frames
|
|
samples="$(awk -v d="$1" -v r="$SAMPLE_RATE" 'BEGIN { printf "%d\n", int(d * r + 0.5) }')"
|
|
frames=$((samples / HOP_SAMPLES))
|
|
printf '%d\n' "$((frames * RVQ_BYTES_PER_FRAME))"
|
|
}
|
|
|
|
# ------------------------------------------------------------------- build --
|
|
build_voice() {
|
|
local voice="$1"
|
|
# The pack dir name is the voice name on the wire: app.py's VOICE_NAME_RE
|
|
# matches it exactly while the engine lowercases names internally, so an
|
|
# uppercase or punctuated name would be reported one way by the façade and
|
|
# another by the engine. Lowercase only, and nothing that could escape
|
|
# $PACKS_DIR.
|
|
[[ "$voice" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]] ||
|
|
die "voice name '$voice' is not a lowercase [a-z0-9._-] name starting with alphanumerics (app.py resolves it verbatim; the engine lowercases it)"
|
|
|
|
local prefix
|
|
prefix="VOICE_PACK_$(printf '%s' "$voice" | tr '[:lower:]' '[:upper:]' | tr -c '[:alnum:]' '_')"
|
|
local source_var="${prefix}_SOURCE" offset_var="${prefix}_OFFSET"
|
|
local duration_var="${prefix}_DURATION" sha_var="${prefix}_SHA256"
|
|
local source="${ARG_SOURCE:-${!source_var:-}}"
|
|
local offset="${ARG_OFFSET:-${!offset_var:-}}"
|
|
local duration="${ARG_DURATION:-${!duration_var:-}}"
|
|
local want_sha="${ARG_SHA:-${!sha_var:-}}"
|
|
|
|
if [ -z "$voice" ] || [ -z "$source" ]; then
|
|
die "no source for '$voice': pass --source or set ${source_var} in $ENV_FILE"
|
|
fi
|
|
[ -n "$offset" ] || die "no crop offset for '$voice': pass --offset or set ${offset_var}"
|
|
[ -n "$duration" ] || die "no crop duration for '$voice': pass --duration or set ${duration_var}"
|
|
is_seconds "$offset" || die "crop offset '$offset' is not a number of seconds"
|
|
is_seconds "$duration" || die "crop duration '$duration' is not a number of seconds"
|
|
[ -f "$source" ] && [ -r "$source" ] || die "source not readable: $source"
|
|
|
|
# === THE GATE. Source seconds 0..10 are forbidden voice material, full stop.
|
|
if ! fcmp "$offset" '>=' "$SOURCE_FORBIDDEN_SECONDS"; then
|
|
die "source-material invariant violated: the crop must start past the first ${SOURCE_FORBIDDEN_SECONDS} s of the source file, and '$voice' asks for ${offset} s into $source. Pick an offset at or past ${SOURCE_FORBIDDEN_SECONDS} s — nothing was written."
|
|
fi
|
|
|
|
local pack_dir="$PACKS_DIR/$voice"
|
|
if [ "$DRY_RUN" -ne 1 ] && [ -e "$pack_dir" ] && [ "$FORCE" -ne 1 ]; then
|
|
die "$pack_dir already exists; nothing was touched. Re-run with --force to retire it to $PACKS_DIR/$TRASH_DIRNAME/ and rebuild."
|
|
fi
|
|
|
|
local source_duration source_bytes source_sha
|
|
source_duration="$(ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 "$source")"
|
|
is_seconds "$source_duration" || die "ffprobe reports no numeric duration for $source: '$source_duration'"
|
|
source_bytes="$(stat -c%s "$source")"
|
|
if ! fcmp "$(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }')" '<=' "$source_duration"; then
|
|
die "crop ends at $(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }') s but $source is only $source_duration s; the crop would come out short"
|
|
fi
|
|
if ! fcmp "$duration" '>=' "$MIN_SECONDS" || ! fcmp "$duration" '<=' "$MAX_SECONDS"; then
|
|
die "crop duration $duration s is outside the ${MIN_SECONDS}-${MAX_SECONDS} s reference window"
|
|
fi
|
|
local crop_end planned_rvq
|
|
crop_end="$(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }')"
|
|
planned_rvq="$(rvq_bytes_for "$duration")"
|
|
if ! fcmp "$planned_rvq" '>=' "$RVQ_MIN_BYTES" || ! fcmp "$planned_rvq" '<=' "$RVQ_MAX_BYTES"; then
|
|
die "a $duration s crop packs to ${planned_rvq} B of .rvq, outside the recorded engine window ${RVQ_MIN_BYTES}-${RVQ_MAX_BYTES} B; lengthen the crop (that window starts at $(awk -v m="$RVQ_MIN_BYTES" -v b="$BYTES_PER_SECOND" 'BEGIN { print m / b }') s) — the source, not the codec, sets this floor"
|
|
fi
|
|
|
|
echo "VOICE $voice"
|
|
echo "SOURCE $source ($source_bytes bytes, ${source_duration} s)"
|
|
echo "CROP ${offset} s .. ${crop_end} s (${duration} s) — past the ${SOURCE_FORBIDDEN_SECONDS} s forbidden region"
|
|
|
|
source_sha="$(sha256sum "$source" | awk '{ print $1 }')"
|
|
if [ -n "$want_sha" ]; then
|
|
is_hex64 "$want_sha" || die "expected sha256 '$want_sha' is not 64 hex digits"
|
|
[ "$source_sha" = "$want_sha" ] ||
|
|
die "source sha256 is $source_sha, expected $want_sha: $source is not the file these offsets were cut from"
|
|
fi
|
|
echo "SHA256 $source_sha"
|
|
|
|
# Where the crop is cut. A dry run keeps its work in a scratch dir it owns;
|
|
# a real build stages INSIDE the pack dir so the final rename is on one
|
|
# filesystem and the pack becomes visible only when it is whole.
|
|
local stage_dir
|
|
if [ "$DRY_RUN" -eq 1 ]; then
|
|
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/voice-pack-$voice.XXXXXX")"
|
|
stage_dir="$WORK_DIR"
|
|
else
|
|
mkdir -p "$PACKS_DIR"
|
|
if [ -e "$pack_dir" ]; then
|
|
local trash
|
|
trash="$PACKS_DIR/$TRASH_DIRNAME/$voice-$(date -u +%Y%m%dT%H%M%SZ)"
|
|
mkdir -p "$(dirname "$trash")"
|
|
mv "$pack_dir" "$trash"
|
|
echo "RETIRED $pack_dir -> $trash (nothing deleted)"
|
|
fi
|
|
mkdir "$pack_dir" || die "cannot create $pack_dir (it may exist now); nothing was overwritten"
|
|
stage_dir="$pack_dir"
|
|
fi
|
|
local stage_wav="$stage_dir/staging-reference.wav"
|
|
|
|
nice -n "$NICE_LEVEL" env OMP_NUM_THREADS="$THREADS" \
|
|
ffmpeg -hide_banner -loglevel error -y \
|
|
-ss "$offset" -t "$duration" -i "$source" \
|
|
-map 0:a:0 -vn -ac 1 -ar "$SAMPLE_RATE" -c:a pcm_s16le "$stage_wav"
|
|
|
|
local wav_rate wav_channels wav_codec wav_duration
|
|
wav_rate="$(ffprobe -v error -select_streams a:0 -show_entries stream=sample_rate -of default=nw=1:nk=1 "$stage_wav")"
|
|
wav_channels="$(ffprobe -v error -select_streams a:0 -show_entries stream=channels -of default=nw=1:nk=1 "$stage_wav")"
|
|
wav_codec="$(ffprobe -v error -select_streams a:0 -show_entries stream=codec_name -of default=nw=1:nk=1 "$stage_wav")"
|
|
wav_duration="$(ffprobe -v error -select_streams a:0 -show_entries stream=duration -of default=nw=1:nk=1 "$stage_wav")"
|
|
is_seconds "$wav_rate" || die "ffprobe reports no numeric sample rate for the crop: '$wav_rate'"
|
|
is_seconds "$wav_duration" || die "ffprobe reports no numeric duration for the crop: '$wav_duration'"
|
|
[ "$wav_rate" = "$SAMPLE_RATE" ] || die "reference.wav came out at ${wav_rate} Hz, not ${SAMPLE_RATE}"
|
|
[ "$wav_channels" = "1" ] || die "reference.wav came out with $wav_channels channels, not mono"
|
|
[ "$wav_codec" = "pcm_s16le" ] || die "reference.wav came out as $wav_codec, not the pcm_s16le the law names"
|
|
if ! fcmp "$wav_duration" '>=' "$MIN_SECONDS" || ! fcmp "$wav_duration" '<=' "$MAX_SECONDS"; then
|
|
die "reference.wav is ${wav_duration} s, outside the ${MIN_SECONDS}-${MAX_SECONDS} s window"
|
|
fi
|
|
echo "REFERENCE ${wav_rate} Hz ${wav_codec} mono, ${wav_duration} s, $(stat -c%s "$stage_wav") bytes"
|
|
|
|
local spk_file="$stage_dir/staging-reference.spk" rvq_file="$stage_dir/staging-reference.rvq"
|
|
local stage_log="$stage_dir/staging-qwen-codec.log"
|
|
local spk_bytes=0 rvq_bytes=0 produced=false
|
|
if [ "$DRY_RUN" -eq 1 ]; then
|
|
if [ -n "$TOOL" ] && [ -x "$TOOL" ]; then
|
|
echo "EXTRACTOR $TOOL (present; --dry-run will not run it)"
|
|
else
|
|
echo "EXTRACTOR not present (${TOOL:-unset}) — --dry-run does not need it"
|
|
fi
|
|
echo "PLANNED reference.spk ${SPK_BYTES} B, reference.rvq $(rvq_bytes_for "$wav_duration") B"
|
|
else
|
|
[ -n "$TOOL" ] && [ -x "$TOOL" ] ||
|
|
die "speaker extractor missing or not executable: ${TOOL:-VOICE_PACK_TOOL unset} (use --dry-run to build everything up to this point)"
|
|
[ -f "$CODEC_MODEL" ] || die "codec model missing: $CODEC_MODEL"
|
|
[ -f "$TALKER_MODEL" ] || die "talker model missing: $TALKER_MODEL"
|
|
# The extractor takes the WAV and writes its outputs next to it with the
|
|
# extension swapped — there is no -o, so the staged name decides where
|
|
# the artifacts land.
|
|
local extractor=("$TOOL" --model "$CODEC_MODEL" --talker "$TALKER_MODEL" -i "$stage_wav")
|
|
if [ -n "$DEVICE" ]; then
|
|
extractor+=(--device "$DEVICE")
|
|
fi
|
|
echo "EXTRACT $(basename "$TOOL") at $ENGINE_COMMIT (device ${DEVICE:-auto}, $THREADS threads)"
|
|
if ! nice -n "$NICE_LEVEL" env OMP_NUM_THREADS="$THREADS" \
|
|
"${extractor[@]}" 2>&1 | tee "$stage_log" >&2; then
|
|
die "$TOOL failed on $stage_wav (its output is in $stage_log); the pack was not published"
|
|
fi
|
|
[ -f "$spk_file" ] || die "$TOOL wrote no $spk_file (see $stage_log)"
|
|
[ -f "$rvq_file" ] || die "$TOOL wrote no $rvq_file (see $stage_log)"
|
|
spk_bytes="$(stat -c%s "$spk_file")"
|
|
rvq_bytes="$(stat -c%s "$rvq_file")"
|
|
local expect_rvq
|
|
expect_rvq="$(rvq_bytes_for "$wav_duration")"
|
|
[ "$spk_bytes" = "$SPK_BYTES" ] ||
|
|
die "reference.spk is $spk_bytes B, expected ${SPK_BYTES} B (enc_dim x f32). The engine changed its speaker encoder — see [[Voice Packs]] before shipping this pack"
|
|
[ "$rvq_bytes" = "$expect_rvq" ] ||
|
|
die "reference.rvq is $rvq_bytes B, expected ${expect_rvq} B for a ${wav_duration} s crop (22 B per 12.5 Hz frame)"
|
|
if ! fcmp "$rvq_bytes" '>=' "$RVQ_MIN_BYTES" || ! fcmp "$rvq_bytes" '<=' "$RVQ_MAX_BYTES"; then
|
|
die "reference.rvq is $rvq_bytes B, outside the recorded engine window ${RVQ_MIN_BYTES}-${RVQ_MAX_BYTES} B — an engine bump? see [[Voice Packs]]"
|
|
fi
|
|
produced=true
|
|
echo "ARTIFACTS reference.spk ${spk_bytes} B, reference.rvq ${rvq_bytes} B"
|
|
fi
|
|
|
|
local built stage_manifest
|
|
built="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
stage_manifest="$stage_dir/staging-manifest.json"
|
|
jq -n \
|
|
--arg voice "$voice" \
|
|
--arg built "$built" \
|
|
--arg builder "scripts/build-voice-pack.sh" \
|
|
--arg env_file "$ENV_FILE" \
|
|
--arg src "$source" \
|
|
--arg src_sha "$source_sha" \
|
|
--argjson src_bytes "$source_bytes" \
|
|
--argjson src_duration "$source_duration" \
|
|
--argjson offset "$offset" \
|
|
--argjson end "$crop_end" \
|
|
--argjson duration "$duration" \
|
|
--argjson forbidden "$SOURCE_FORBIDDEN_SECONDS" \
|
|
--argjson rate "$SAMPLE_RATE" \
|
|
--argjson wav_duration "$wav_duration" \
|
|
--arg wav_codec "$wav_codec" \
|
|
--argjson wav_channels "$wav_channels" \
|
|
--argjson wav_bytes "$(stat -c%s "$stage_wav")" \
|
|
--arg tool "${TOOL:-unset}" \
|
|
--arg repo "$ENGINE_REPO" \
|
|
--arg commit "$ENGINE_COMMIT" \
|
|
--arg codec "$CODEC_MODEL" \
|
|
--arg talker "$TALKER_MODEL" \
|
|
--arg device "$DEVICE" \
|
|
--argjson produced "$produced" \
|
|
--argjson spk_bytes "$spk_bytes" \
|
|
--argjson rvq_bytes "$rvq_bytes" \
|
|
--argjson planned_rvq "$(rvq_bytes_for "$wav_duration")" \
|
|
--argjson dry_run "$DRY_RUN" \
|
|
'{
|
|
voice: $voice,
|
|
kind: "voice-pack",
|
|
built_utc: $built,
|
|
builder: $builder,
|
|
tunables: $env_file,
|
|
dry_run: ($dry_run == 1),
|
|
source: {
|
|
path: $src,
|
|
sha256: $src_sha,
|
|
bytes: $src_bytes,
|
|
duration_seconds: $src_duration
|
|
},
|
|
crop: {
|
|
offset_seconds: $offset,
|
|
end_seconds: $end,
|
|
duration_seconds: $duration
|
|
},
|
|
reference: {
|
|
file: "reference.wav",
|
|
codec: $wav_codec,
|
|
sample_rate: $rate,
|
|
channels: $wav_channels,
|
|
duration_seconds: $wav_duration,
|
|
bytes: $wav_bytes
|
|
},
|
|
artifacts: {
|
|
produced: $produced,
|
|
tool: $tool,
|
|
engine_repo: $repo,
|
|
engine_commit: $commit,
|
|
device: $device,
|
|
codec_model: $codec,
|
|
talker_model: $talker,
|
|
spk: {file: "reference.spk", bytes: $spk_bytes},
|
|
rvq: {file: "reference.rvq", bytes: $rvq_bytes, planned_bytes: $planned_rvq}
|
|
},
|
|
invariant: {
|
|
name: "source-material invariant",
|
|
rule: "a voice crop MUST start past the first 10 seconds of the source file",
|
|
forbidden_source_seconds: $forbidden,
|
|
crop_offset_seconds: $offset
|
|
}
|
|
}' >"$stage_manifest"
|
|
|
|
# Assert the gate a second time, FROM the manifest, next to the bytes it
|
|
# describes. Anything short of a pass leaves the staged pack unpublished.
|
|
local recorded_offset recorded_sha
|
|
recorded_offset="$(jq -r '.crop.offset_seconds' "$stage_manifest")"
|
|
recorded_sha="$(jq -r '.source.sha256' "$stage_manifest")"
|
|
jq -e --argjson min "$SOURCE_FORBIDDEN_SECONDS" \
|
|
'(.crop.offset_seconds | type == "number") and .invariant.forbidden_source_seconds == $min and (.crop.offset_seconds >= $min) and (.invariant.name == "source-material invariant")' \
|
|
"$stage_manifest" >/dev/null ||
|
|
die "manifest assertion failed: .crop.offset_seconds ($recorded_offset) does not clear the ${SOURCE_FORBIDDEN_SECONDS} s source-material invariant"
|
|
[ "$recorded_sha" = "$source_sha" ] ||
|
|
die "manifest assertion failed: .source.sha256 ($recorded_sha) is not the hash of $source ($source_sha)"
|
|
jq -e '(.voice | type == "string") and (.reference.sample_rate | type == "number") and (.reference.duration_seconds | type == "number") and (.artifacts.engine_commit | length == 40)' \
|
|
"$stage_manifest" >/dev/null ||
|
|
die "manifest assertion failed: voice / reference / engine provenance fields are not all present"
|
|
echo "MANIFEST crop offset $recorded_offset s clears the ${SOURCE_FORBIDDEN_SECONDS} s invariant"
|
|
|
|
if [ "$DRY_RUN" -eq 1 ]; then
|
|
echo "MANIFEST_BEGIN"
|
|
cat "$stage_manifest"
|
|
echo "MANIFEST_END"
|
|
echo "PLAN would write $pack_dir/{reference.wav,reference.spk,reference.rvq,manifest.json}"
|
|
echo "DRYRUN $voice: inputs valid, crop cut and probed, extractor not run"
|
|
else
|
|
# reference.wav is the gate the shim reads, so it lands last: the voice
|
|
# name never exists on disk in a state the artifacts are not behind it.
|
|
mv "$stage_log" "$pack_dir/qwen-codec.log"
|
|
mv "$spk_file" "$pack_dir/reference.spk"
|
|
mv "$rvq_file" "$pack_dir/reference.rvq"
|
|
mv "$stage_manifest" "$pack_dir/manifest.json"
|
|
mv "$stage_wav" "$pack_dir/reference.wav"
|
|
echo "BUILT $pack_dir (voice '$voice': ${wav_duration} s, spk ${spk_bytes} B, rvq ${rvq_bytes} B)"
|
|
echo "VOICE_PACK_BUILT $voice $pack_dir"
|
|
fi
|
|
|
|
}
|
|
|
|
if [ "$BUILD_ALL" -eq 1 ]; then
|
|
[ -z "$VOICE" ] || die "--all takes no voice name (got '$VOICE'); the list comes from VOICE_PACKS"
|
|
for flag in "$ARG_SOURCE" "$ARG_OFFSET" "$ARG_DURATION" "$ARG_SHA"; do
|
|
[ -z "$flag" ] || die "--all reads every crop from $ENV_FILE; --source/--offset/--duration/--sha256 would apply one value to all of them"
|
|
done
|
|
read -r -a voices <<<"$VOICE_PACKS"
|
|
[ "${#voices[@]}" -gt 0 ] || die "VOICE_PACKS is empty in $ENV_FILE"
|
|
for voice in "${voices[@]}"; do
|
|
build_voice "$voice"
|
|
done
|
|
else
|
|
[ -n "$VOICE" ] || { usage >&2; die "no voice name given"; }
|
|
build_voice "$VOICE"
|
|
fi
|