Compare commits

..
Author SHA1 Message Date
Frank 2427fc448f docs(voice-packs): runbook, index link, and the log entry
The page a new operator needs: what the three required files are and who
reads each one, how a pack is built (with the extractor's usage text
verbatim, because the flags are not guessable and there is no -o), the
10-second gate and where it fires twice, how to add a fourth voice
including the warm hook that registers a pack without restarting the
engine, and the engine-bump procedure for when the artifact sizes move.
Plus the host census as of 2026-10-18 — the trial packs carry no
artifacts and their manifest is the trial's own schema, so the builder's
first real run is a --force migration — and the three gaps left open on
purpose: no curated ICL transcript, no 16 kHz variant, no loudness pass.

Linked from the Dankclaw services list; log entry appended oldest-first.
2026-10-18 17:25:34 -07:00
Frank 3824d03ac5 Merge remote-tracking branch 'origin/main' into feat/voice-pack-builder 2026-10-18 17:18:31 -07:00
Frank 042003a969 feat(voice-packs): tracked tunables, the bats seam, and its wiring
scripts/voice-packs.env carries the decisions worth a diff and nothing
else: the pack list, each voice's source file, crop offset, crop length
and pinned source digest, the extractor path with its engine repo and
commit, and the artifact arithmetic. Paths and numbers only — the pack
directories themselves stay host state under /home/dank/voices.

tests/voice_pack_unit.bats pins behaviour, from inputs to observable
output: a crop inside the first ten seconds exits nonzero naming the
invariant and creates nothing, a legal one builds the whole pack, the
reference comes out 24 kHz mono inside 18-25 s (ffprobe, on a locally
generated tone), the manifest round-trips through jq with offsets and a
sha256 that matches the source bytes, a re-run without --force leaves
every byte untouched, --force retires rather than deletes, --dry-run
needs no extractor at all, and a stub extractor that changed its artifact
size cannot publish a voice. No engine binary, no GPU, no network, no
fleet host: qwen-codec is stood in by a stub with the real tool's I/O
shape, the way frank_tts_unit.bats drives its backends.

test-changed.sh routes both new paths to that seam instead of letting
them land on the generic scripts/* syntax fallback.
2026-10-18 17:18:30 -07:00
Frank 2242fcfec0 feat(voice-packs): builder — crop, pinned artifacts, manifest, 10 s gate
Turns a source recording plus a crop offset into the pack frank-tts
resolves by name: reference.wav (24 kHz mono pcm_s16le, 18-25 s), the
engine's own reference.spk / reference.rvq made from it, and manifest.json
provenance (source path + sha256, crop start/end, sample rate, duration,
tool, engine commit, voice name).

The extractor is qwen-codec from the pinned qwentts.cpp tree, and the
flags are its own usage text, not a guess: --model <codec gguf>, --talker
<base talker gguf>, -i <wav>, optional --device; there is no -o, so it
names both artifacts next to the input with the extension swapped, which
is why the pack's WAV is staged under its final stem.

The 10-second source-material invariant is enforced, not annotated: an
offset inside the forbidden region exits 1 before a directory is created,
and the gate is asserted a second time from the manifest that was just
written, together with the recorded sha256. Artifact sizes are checked
against the codec's own arithmetic (8,192 B x-vector; 22 B per 12.5 Hz
frame), so an engine bump fails the build instead of shipping a pack the
engine rejects at synthesis time — and the crop is refused up front when
it is too short to fill the .rvq window.

reference.wav is the file the façade gates on, so it lands last, after
the artifacts and the manifest: a voice name never exists on disk without
its artifacts behind it. Nothing is ever deleted — an existing pack is
refused unless --force, which retires it whole to .AGENTTRASH/ — and
--dry-run runs every step except the extractor, so the seam can test the
gate without the engine.
2026-10-18 17:18:21 -07:00
dank e13f875c94 Merge pull request 'docs(tts): record the measured CPU engine A/B behind frank-tts' (#10) from docs/tts-engine-trials into main 2026-10-18 17:15:00 -07:00
Frank 1d082a7f1d Merge remote-tracking branch 'origin/main' into docs/tts-engine-trials
# Conflicts:
#	serverAppConfigs-docs/log.md
2026-10-18 16:54:13 -07:00
Frank 5c8ac208a0 Merge remote-tracking branch 'origin/main' into docs/tts-engine-trials 2026-10-18 16:51:01 -07:00
Frank dbc2c67bcb docs(tts): record the arm A engine decision and the loopback-publish law
Frank heard every audition pair and passed it, so likeness stopped being the
tiebreak and the engine was picked on the live-output criterion: arm A
qwentts.cpp 5916bf7, resident tts-server, 1.7 B talker, llama-tts-cli kept as
the rollback. Records the discriminators (first audio byte, warm RTF, spread),
the three consequences that are configuration (boot-warm every voice, no cpuset
pin, WAV-only registration), the two follow-ups left open on purpose, and the
corrected addressing law: a host-loopback publish is unreachable from a bridge
container, so the pair shares one quadlet network.
2026-10-18 16:51:01 -07:00
dank 1eb8b2199a Merge pull request 'feat(open-notebook): podcast pipeline on Dankclaw (blocked on frank-tts)' (#12) from feat/open-notebook-podcast into main 2026-10-18 16:50:47 -07:00
dank 0e4bb327da Merge pull request 'docs: attribution footer for machine-opened issues and PRs' (#11) from chore/pr-attribution-trailer into main 2026-10-18 16:46:00 -07:00
Frank 114dd7a93e fix(open-notebook): size the TTS clock outside frank-tts's own ceiling
The env file first landed with ESPERANTO_TTS_TIMEOUT equal to SHIM_TIMEOUT=600.
Equality is a race: both budgets expire at the same instant, so the case the
number existed to prevent — the worker abandoning a synthesis frank-tts is still
finishing — is exactly what happens on the tail of a long line. Nested budgets
go inside-out, which is the rule scripts/frank-tts.env states for its own
LLAMA_TIMEOUT; the client waits one round-trip longer than the server, so a
frank-tts give-up arrives labelled as frank-tts's error. 660.

The seam's assertion moves from equality to `>` for the same reason, and now
also holds ESPERANTO_LLM_TIMEOUT under upstream's 600 s web-UI patience — above
it the failure surfaces as a UI timeout with no name on it. The scriptwriting
number itself stays at the app's 180 s default: frank-zealot answers an outline
or a transcript well inside it, which corrects the earlier commit message's
claim that both timeouts were raised past upstream's 300 s.
2026-10-18 16:31:49 -07:00
Frank e2393bacad docs(open-notebook): runbook, episode-destination interim, and the log entry
Page registered in the vault index (the docs-completeness seam fails an
unregistered page) and logged in the bundle change log.

What the page owes the reader: the provider contract (frank-zealot for scripts
with a /v1 base URL, frank-tts for voices with 1-4 speakers mapped to the
narrator1/2/3 voice packs typed as free text), why Test Connection and Discover
Models fail cosmetically against frank-tts rather than being a fault, the
encryption key as the precondition for saving any credential, and the honest
blocker — frank-tts is not deployed and nothing listens on 7772, so a generation
run writes its script and fails on the first voice.

Episode destination is stated as interim and cites Gitea issue #7 (human-only,
agents must not work it): the app keeps its named volume, one bind mount puts
finished MP3s under /home/dank/media/podcasts, and deleting that one line is the
whole rollback. Rollback section also covers the full undo, including that the
podman network only disappears once the units are gone.
2026-10-18 16:27:03 -07:00
Frank ff11c37ee0 test(open-notebook): seam for the units, the env guards, and the wiring
Structure only: what the units declare and how the scripts route them, never how
the app behaves inside a container. 19 tests.

The ones carrying real weight: the app unit has no publishable 8000 anywhere and
the database unit publishes nothing at all; the API remap off host 5055 is only
correct if API_URL names the same host port, so the seam parses the PublishPort
and compares; each unit reads the repo tunable file before the host secret file,
because podman's last-match-wins makes the order load-bearing; both health
probes are argument-only (a shell string there is a unit that starts dead); the
database password appears only as an env reference, never in Exec=; and the
episode host bind appears exactly once, since a second mount would silently
shadow the first.

The deployer assertions read the functions themselves: the gate URL path is
/health on the remapped port and must not name 5055, and the seed function
contains the blank-secret refusal plus the no-clobber test. No test asserts a
made-up endpoint or a cloud URL, and nothing touches the network.
2026-10-18 16:27:03 -07:00
Frank 0fe23c5233 build(scripts): route open-notebook paths to its own seam
The case has to sit where it does: the dankclaw/*.container glob's `*` crosses
`/`, so placed below it, dankclaw/open-notebook/quadlet/open-notebook.container
matched that generic case and ran the open-webui/pihole/searxng seams instead of
its own. Same reason the env path is grouped here — scripts/*.env would otherwise
send it to the generic env seams.
2026-10-18 16:27:03 -07:00
Frank 716eefef5d feat(homepage): open-notebook card, hidden until the service is deployed
Internal-only: port 8502 direct, no vhost field, no Caddy change — exposure
stays behind an explicit decision, and the app carries its own password.

urlStatus=hidden is deliberate: the homepage probe runs on the gateway and the
pair is not deployed, so a live card would render dead on day one. Flipping it
to live is the only homepage change the deploy needs; the seam asserts the
hidden state while the service is repo-only and requires the required card
fields either way.

mono-on.svg follows the mono-* icon set the sibling cards use, so the card does
not fall back to the generic dot.
2026-10-18 16:26:52 -07:00
Frank 8816a9d498 feat(deploy): wire open-notebook through both deployers, gated on secrets
Registration is load-bearing, not cosmetic: deploy-plan.py refuses any service
it does not know, so SERVICE_ORDER + DANKCLAW_SERVICES are what let `--only` and
`--auto` address the pair at all; the dispatch case in deploy-remote.sh runs it;
the requirement in check_prerequisites gates a manual deploy; and the dispatch
regex in deploy.sh's host-mode branch decides whether a changed path dispatches
a service or falls through to "no service mapped" — an omission there is silent,
which is exactly how the AGENTS.md deployer-registration law was earned.

The deploy seeds /home/dank/Containers/open-notebook/env (0600) from the host
env and refuses to start the pair while OPEN_NOTEBOOK_ENCRYPTION_KEY or
OPEN_NOTEBOOK_PASSWORD is blank: without the key no provider credential saves
and without the password the API — including the episode-audio route — is open
to the LAN with no auth. A seeded file is never rewritten, so re-deploying
cannot clobber a rotated key, which would silently make every stored credential
undecryptable. SURREAL_PASS is derived from SURREAL_PASSWORD in the same write.

The gate names paths that answer, not invented ones: the database is judged by
its own container health status (/surreal is-ready, verified against the pinned
digest) and the app by wait_for_health on /health at the published 5056 mapping —
the route is upstream's and is on the auth middleware's exclude list, so it
stays 200 after the password is set. 180 s covers Next.js build + startup and a
degraded-but-listening first boot, matching the sibling LLM deploys.

Both digest pins are checked against local storage before restart, because
Pull=never means a missing image would otherwise surface as a unit stuck in
"Start request queued" (the 203/EXEC trap's sibling). The interim episode
directory is created before the app starts: a missing bind source makes podman
create it as root-owned and the app's mkdir would then fail.
2026-10-18 16:26:52 -07:00
Frank 032229c1e2 feat(open-notebook): env contract — CPU guard, credential store, auth posture
The two knobs that keep the box alive: TTS_BATCH_SIZE=1 and
OPEN_NOTEBOOK_WORKER_MAX_TASKS=1. Upstream ships 5 x 5; frank-tts spawns one
~8 GiB, 8-thread engine process per synthesis request on a machine that also
serves two LLMs, so a five-way fan-out is a self-inflicted shutdown. The worker
knob is real, not decorative: the image's supervisord expands
${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5} into the worker command
(supervisord.conf:25).

Both ESPERANTO_* timeouts go above upstream's 300 s defaults, sized for CPU
synthesis (SHIM_TIMEOUT=600, LLAMA_TTS_THREADS=8) — one episode is dozens of
per-line /audio/speech calls and a timeout kills the episode, not the line.

API_URL is forced for the 5055 -> 5056 remap; SURREAL_URL addresses the database
by container name; CORS_ORIGINS is origin-form (scheme://host:port), the form
the app compares against. OPEN_NOTEBOOK_PASSWORD stays absent here on purpose:
the app treats a set-but-empty value as auth-on.

Secrets stay placeholders in the tracked file and in dankclaw/.env.example, which
lists SURREAL_PASSWORD rather than SURREAL_PASS — the deployer derives the
surreal CLI's spelling from it when it seeds, so the app and the database cannot
be handed different values.
2026-10-18 16:26:43 -07:00
Frank 09b0badaf5 feat(open-notebook): Quadlet pair — app + SurrealDB on a private bridge
Two containers, because that is what upstream runs (docker-compose.yml:6,13):
the app image under supervisord, plus SurrealDB. The `v1-latest-single` tag is
not in play — it bundles its own database, is deprecated upstream, and would
give the host a second hidden DB.

Both images are digest-pinned with Pull=never and a documented one-time pull,
the frank-dragoon posture: a Quadlet on :latest re-resolves on the first restart
after an unrelated daemon-reload, and the failure is silent.

Port map paid for by what is already on the host: seerr holds 127.0.0.1:5055
(since 2026-10-14, live `ss -ltn` agrees), so the API leaves on 5056 and the env
file has to set API_URL to match, because the frontend derives the API URL from
the window origin when it is unset. 8502 publishes on 0.0.0.0 — the service
carries its own password and the episode-audio route is unauthenticated. 8000 is
published nowhere, where upstream exposes it on loopback with root credentials.

The app joins dankclaw_default first and the private systemd-open-notebook
second: it reaches the published fleet endpoints through the former
(host.containers.internal, the arrangement Prometheus is proven against) and
finds its database by container name through the latter. Generated --network
flags keep file order, so the tunable file is listed before the host secret file
in both units — podman applies last-match-wins.

No UserMapping: the app runs as root in its namespace (no USER in the Dockerfile)
and writes only paths owned by that root. Exec= carries no database credential —
`exec start --user --pass` puts it in world-readable argv; the image's own
entrypoint passes it on stdin, so only the container env holds it.
2026-10-18 16:26:37 -07:00
Frank 0e7936f94e docs(tts): record the measured CPU engine A/B behind frank-tts
Both candidate :8880 engines were measured on Dankclaw's CPU (no GPU,
no container, nothing deployed). arm A qwentts.cpp 5916bf7 resident at
RTF 0.55-0.57 warm / 0.67 ICL / 1.65 cold; arm B mainline llama-tts at
0.93-0.99 with a +-0.25 spread. Records the four configuration laws the
trial produced (boot-warm every voice, .spk+.rvq replay is byte-identical,
streaming != buffered, /v1/audio/voices rejects a lone spk_b64), the
likeness measurement's resolution limit, and re-keys the stale
8890/voiceshim shape on that page to frank-tts on 7772. The engine pick
stays open: the likeness metric cannot rank clones, so the audition clips
decide.
2026-10-18 16:19:51 -07:00
20 changed files with 2367 additions and 25 deletions
+20
View File
@@ -79,3 +79,23 @@ MARRETA_APP_KEY=changeme
# account on first boot from these values. # account on first boot from these values.
MARRETA_ADMIN_EMAIL=admin@franksdank.cloud MARRETA_ADMIN_EMAIL=admin@franksdank.cloud
MARRETA_ADMIN_PASSWORD=changeme MARRETA_ADMIN_PASSWORD=changeme
# open-notebook (podcast generator; internal-only, no vhost).
# deploy-remote.sh seeds /home/dank/Containers/open-notebook/env from these
# values (only if that file does not already exist) and refuses to start the
# units while any of them is blank: a blank OPEN_NOTEBOOK_PASSWORD means
# authentication OFF, and a blank encryption key means no provider credential
# can be saved. The seed writes SURREAL_PASS too — the spelling the `surreal`
# CLI reads — from SURREAL_PASSWORD, so the app and the database cannot be
# given different values.
# OPEN_NOTEBOOK_ENCRYPTION_KEY: encrypts the stored AI provider keys at rest.
# Generate: openssl rand -hex 32
# Changing it after credentials are saved makes them undecryptable.
OPEN_NOTEBOOK_ENCRYPTION_KEY=changeme
# UI/API password. Every API route — including the episode-audio download the
# app serves without a session — is open to the LAN while this is unset.
OPEN_NOTEBOOK_PASSWORD=changeme
# SurrealDB root password (the root user NAME comes from
# scripts/open-notebook.env, so this file holds only the secret).
# Generate: openssl rand -base64 32
SURREAL_PASSWORD=changeme
@@ -0,0 +1,84 @@
# Quadlet unit: open-notebook's SurrealDB (rootless, boot-persistent).
#
# Install path on dankclaw: ~/.config/containers/systemd/open-notebook-surrealdb.container
# Unit name: open-notebook-surrealdb.service (container name: open-notebook-surrealdb)
#
# WHAT IT IS. open-notebook's only datastore: notebooks, notes, sources,
# provider credentials and podcast episodes all live here. The app talks to it
# over the WebSocket RPC on the private bridge (SURREAL_URL in
# scripts/open-notebook.env), which is why the container NAME is part of the
# contract — renaming it breaks the app's database URL, not just this file.
#
# IMAGE. `surrealdb/surrealdb:v2` pinned by manifest-list digest, resolved from
# the registry on 2026-10-18 (that digest is SurrealDB 2.7.0, image built
# 2026-09-23; `v2` itself is a MOVING tag, so the digest — not the tag — is what
# makes this reproducible). Upstream open-notebook's own compose file pins the
# same `v2` tag, and its Dockerfile copies /surreal out of the same image for
# the retired single-container variant, so v2 is the version this app is built
# against. Pull=never + the documented pull step (docs: [[Open Notebook]]):
#
# podman pull docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723
#
# NO PUBLISHED PORT. 8000 is deliberately not published: the app reaches it by
# name on systemd-open-notebook, and a published 8000 would put the RPC with
# the root credentials on every interface (upstream publishes 127.0.0.1:8000
# "purely for local debugging" — a `podman exec` probe is enough here). The
# binary binds 0.0.0.0:8000 INSIDE its own namespace via the image's
# SURREAL_BIND env; without that image default it would bind loopback only and
# the app could never connect, so no --bind flag belongs here.
#
# CREDENTIALS, AND WHY THEY ARE NOT IN THE COMMAND LINE. The `surreal start`
# CLI reads its options from the environment (`--username` is [env:
# SURREAL_USER=], `--password` is [env: SURREAL_PASS=] — verified against the
# pinned image's own `start --help`), so the credentials arrive through the two
# EnvironmentFile lines and the argv stays clean. Passing `--pass` instead —
# what upstream's compose does — would put the database password in
# /proc/<pid>/cmdline, readable by any `ps` on the host. Note the two
# spellings: the DATABASE reads SURREAL_PASS, the APP reads SURREAL_PASSWORD,
# and scripts/open-notebook.env declares both because they must hold the same
# value; the deployer writes that pair when it seeds the host secret file.
#
# NO USER= OVERRIDE. The image runs as uid 65532 (Chainguard glibc-dynamic
# base). With a NAMED volume that works unmodified: podman creates the volume's
# backing directory owned by the container's user (measured with this digest —
# the host directory lands on the mapped subuid and the store starts clean),
# which is exactly why upstream's `user: root` line is not repeated here: that
# line is commented in upstream as being "required for bind mounts", and this
# unit has none. The consequence is that the DB files are NOT readable by dank
# without `podman unshare` — accepted for a datastore nobody edits by hand;
# episode AUDIO is a host directory for the opposite reason (see the app unit).
#
# HEALTH. The base image is distroless: no shell, no curl, no wget, so no
# CMD-SHELL probe and no wget/curl probe is possible here (same class of unit
# as loki, whose probe had to move into the image). SurrealDB ships its own
# probe instead — `surreal is-ready` connects to ws://localhost:8000 by default
# and needs no credentials — verified against this digest: it prints `OK` and
# exits 0 while the server is up. Argument-only HealthCmd, like every unit on
# this host, because the quadlet generator splits it on spaces.
[Unit]
Description=SurrealDB (open-notebook datastore)
After=network-online.target
Wants=network-online.target
[Container]
ContainerName=open-notebook-surrealdb
Image=docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723
Pull=never
Network=open-notebook.network
EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Exec=start --log info rocksdb:/mydata/mydatabase.db
Volume=open-notebook-db:/mydata
HealthCmd=/surreal is-ready
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=20s
[Service]
Restart=always
RestartSec=5
[Install]
WantedBy=default.target
@@ -0,0 +1,101 @@
# Quadlet unit: open-notebook (rootless, boot-persistent).
#
# Install path on dankclaw: ~/.config/containers/systemd/open-notebook.container
# Unit name: open-notebook.service (container name: open-notebook)
#
# WHAT IT IS. The fleet's podcast generator: notes and sources go in, a
# two-voice (up to four) episode MP3 comes out. One image runs THREE processes
# under supervisord — the FastAPI API on 5055, the background worker that
# writes and speaks the episode, and the Next.js UI on 8502 — so this one unit
# owns the whole app, and the frontend refuses to start until the API answers
# (the image's own wait-for-api.sh). Runbook: [[Open Notebook]].
#
# IMAGE. `lfnovo/open_notebook:v1-latest` pinned by manifest-list digest,
# resolved from the registry on 2026-10-18; it is the tag upstream's own
# docker-compose.yml pins for the multi-container shape. Do NOT reach for the
# `-single` tags: that variant bundles SurrealDB inside the app image, is
# deprecated upstream, and would give this host two databases. Pull=never, so
# the pin is fetched by the documented one-time step (docs: [[Open Notebook]]),
# which the deployer also runs when the image is absent:
#
# podman pull docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212
#
# NETWORKS, in this order on purpose. The generated run line keeps file order,
# so the DEFAULT ROUTE rides dankclaw_default: that is the network where this
# host has measured that a bridge container can reach a host-namespace
# listener through the gateway name (the Prometheus scrape of frank-zealot is
# the proof — see [[Container Network Namespaces]]), and frank-tts lives there
# too, so the TTS credential can be `http://frank-tts:7772/v1`. The private
# systemd-open-notebook bridge is second and carries only the database; its
# traffic is on-link, so losing the default route costs it nothing.
#
# PUBLISHED PORTS, and the 5055 collision. Upstream publishes 8502 and 5055,
# and 5055 is exactly the port seerr already holds on this host
# (`PublishPort=127.0.0.1:5055:5055`; `ss -ltn` on Dankclaw 2026-10-18 shows
# 127.0.0.1:5055 taken). So the API leaves the container on host port 5056.
# That forces one setting: the frontend auto-detects the API as
# `<host-you-opened>:5055`, so API_URL in scripts/open-notebook.env must name
# the remapped port explicitly, or the UI loads and every call fails. Both
# ports are published on all interfaces because the consumer is a browser on
# the LAN — the fleet's browser-facing services (open-webui, comfyui, gitea)
# bind the same way, while loopback-only is reserved for services whose user
# path is the guard chain. This service has NO vhost and no Caddy route: it is
# internal-only until it is guarded, and its own password
# (OPEN_NOTEBOOK_PASSWORD) is what stands in front of it. That password is not
# decoration: with it unset the API answers everyone on the LAN, including the
# unauthenticated episode-audio download.
#
# VOLUMES. `/app/data` is a named volume: uploads, the LangGraph SQLite
# checkpoints and the extraction caches the image keeps under it
# (UV_CACHE_DIR, HF_HOME) are app state nobody browses. The ONE exception is
# finished audio, which is bind-mounted out of the container at
# /home/dank/media/podcasts. That mount is interim and reversible: episode
# destination is an open human-only decision (Gitea issue #7, label
# `human-only` — agents must not work it), and until it lands, episodes are
# ordinary files on the host. DATA_FOLDER is hard-coded `./data` in the app
# (WORKDIR /app) and the DB stores audio paths RELATIVE to data/podcasts, so
# covering that one subdirectory is enough and removing the line changes
# nothing else. The image declares no USER, so the app runs as container root,
# which in this host's rootless mapping (0 -> 1000, measured on this host —
# see [[Seerr]]) writes those episode files as dank.
#
# HEALTH. `GET /health` on the API returns {"status":"healthy"} and is in the
# password middleware's excluded-path list, so the probe stays honest after the
# password is set (both read from api/main.py upstream). The image installs
# curl and ffmpeg in the runtime layer, so the probe needs no shell — kept
# argument-only for the generator's space-splitting. Note what this probe does
# NOT prove: the route returns a static body, so it says the API serves, not
# that the database answers. The DB has its own probe in its unit, and the
# worker's failure to reach the database shows up as failed jobs, not as an
# unhealthy container.
[Unit]
Description=open-notebook (podcast generator: API, worker, web UI)
After=network-online.target open-notebook-surrealdb.service
Wants=network-online.target open-notebook-surrealdb.service
[Container]
ContainerName=open-notebook
Image=docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212
Pull=never
Network=dankclaw_default
Network=open-notebook.network
PublishPort=0.0.0.0:8502:8502
PublishPort=0.0.0.0:5056:5055
EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env
EnvironmentFile=/home/dank/Containers/open-notebook/env
Volume=open-notebook-data:/app/data
Volume=/home/dank/media/podcasts:/app/data/podcasts
HealthCmd=curl -fsS http://127.0.0.1:5055/health
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=60s
[Service]
Restart=always
RestartSec=10
TimeoutStartSec=900
[Install]
WantedBy=default.target
@@ -0,0 +1,34 @@
# Quadlet network: private bridge for the open-notebook pair (rootless).
#
# Install path on dankclaw: ~/.config/containers/systemd/open-notebook.network
# Unit name: open-notebook-network.service (podman network name:
# systemd-open-notebook — quadlet prefixes "systemd-", so probe the runtime
# name, never the filename; see [[Container Network Namespaces]]).
#
# WHY A PRIVATE BRIDGE. The app reaches its database by container name
# (SURREAL_URL=ws://open-notebook-surrealdb:8000/rpc), and container DNS names
# resolve only on the network that owns them. Putting SurrealDB here instead of
# on dankclaw_default keeps the datastore reachable ONLY to the container that
# owns it: 8000 is not published (an unpublished port is refused outright from
# every other namespace anyway), and nothing else on the fleet-wide network can
# dial it with the root credentials. Same reasoning as the gitea and authentik
# database pairs.
#
# Kernel bridge, not slirp4netns: the 2026-08-22 open-webui incident (a dead
# per-container userspace helper left a container that accepted TCP and reset
# every request for two days, logging nothing). A bridge has no such helper.
#
# Subnet registry on dankclaw (rootless):
# 10.88.0.0/16 default "podman" bridge
# 10.89.0.0/24 gitea (systemd-gitea)
# 10.89.1.0/24 compose stack (dankclaw_default)
# 10.89.2.0/24 searxng (systemd-searxng)
# 10.89.3.0/24 open-webui (systemd-open-webui)
# 10.89.4.0/24 open-notebook (systemd-open-notebook) <- this file
# Pinned so the container IPs stay deterministic across reboots, like the
# other pinned nets; 10.89.4.0/24 was the next unclaimed /24 in this registry.
[Network]
Driver=bridge
Subnet=10.89.4.0/24
Gateway=10.89.4.1
+483
View File
@@ -0,0 +1,483 @@
#!/usr/bin/env bash
# build-voice-pack.sh — turn a SOURCE recording plus a CROP OFFSET into a
# deployable frank-tts voice pack: the reference WAV, the engine's own speaker
# artifacts made from it, and manifest.json provenance.
#
# THE PACK. $PACKS_DIR/<voice>/ carries the three members the pack law in
# scripts/frank-tts.env names:
#
# reference.wav the crop: 24 kHz mono PCM (pcm_s16le), 18-25 s of clean
# single-speaker audio. This is what makes a voice NAME real
# (dankclaw/voiceservice/app.py lists and resolves exactly
# this file), so it is written LAST, once everything else in
# the pack has been verified.
# reference.spk the engine's x-vector: enc_dim 2048 f32 = 8,192 B.
# reference.rvq the engine's RVQ code stream: 16 codebooks x 11 bits = 22 B
# per 12.5 Hz frame = 22 * floor(samples / 1920).
# manifest.json provenance: source path and sha256, crop start/end, sample
# rate, duration, the tool + engine commit that made the
# artifacts, and the voice name.
#
# Both artifacts come from the pinned engine build's speaker extractor — the
# `qwen-codec` tool of qwentts.cpp at VOICE_PACK_ENGINE_COMMIT — run on the pack's
# own WAV, and the extractor names them by swapping the input's extension, so
# they land beside reference.wav with nothing to rename. Required, never a
# replacement for the WAV: a voice another runtime cannot reuse is not saved.
#
# THE 10-SECOND GATE — THE REASON THIS SCRIPT EXISTS. Project-wide source-material
# invariant: the first 10 s of any sample file used to generate voices is
# forbidden material. An offset that starts inside that region is refused with a
# nonzero exit BEFORE anything is created — no pack dir, no WAV, no artifact — and
# the same gate is asserted a second time from the manifest that was just written
# (plus the recorded sha256 against the source it claims). A pack carrying a
# forbidden crop therefore cannot survive a build.
#
# THE ARTIFACT SIZES ARE CHECKED, NOT ASSUMED. They are a property of the codec
# config inside the GGUF, so an engine bump that changes enc_dim or the codebook
# count fails the build instead of shipping a pack the engine will reject at
# synthesis time. Note the interaction with the WAV window: an 18.0 s crop is a
# legal reference.wav but only 22 * 225 = 4,950 B of .rvq, under the recorded
# 5,456 B floor, so the effective crop window is 19.84-25 s. That is a real
# constraint and the builder says so up front rather than failing after a build;
# serverAppConfigs-docs/Dankclaw/Voice Packs.md covers what to do on an engine bump.
#
# WHERE IT RUNS: the host that holds the engine build (Dankclaw). The extractor is
# a CPU ggml binary — no GPU, no container, no fleet endpoint, no network. ffmpeg
# does the crop; every spawned process runs `nice -n 19` so a pack build never
# competes with the text endpoints on that box.
#
# IT NEVER DELETES. An existing pack is refused unless --force, and --force moves
# the whole old directory aside into $PACKS_DIR/.AGENTTRASH/<voice>-<UTC>/ rather
# than overwriting or unlinking a single byte.
#
# Usage:
# scripts/build-voice-pack.sh [OPTIONS] VOICE PACKS_DIR
# scripts/build-voice-pack.sh [OPTIONS] --all PACKS_DIR
#
# Options (anything not given comes from scripts/voice-packs.env):
# --env FILE tunables file (default scripts/voice-packs.env)
# --source FILE source recording for this voice
# --offset SEC crop start, in seconds into the source (must be >= 10)
# --duration SEC crop length; reference.wav comes out this long
# --sha256 HEX expected sha256 of the source file
# --tool PATH the pinned speaker extractor (qwen-codec)
# --all build every voice in VOICE_PACKS, skipping none
# --dry-run everything except the extractor: validate, cut the crop,
# probe it, print the plan and the manifest it would write
# --force retire an existing pack to .AGENTTRASH/ and rebuild
# -h, --help this text
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ENV_FILE="$REPO_ROOT/scripts/voice-packs.env"
SOURCE_FORBIDDEN_SECONDS=10 # the source-material invariant; not a knob
WORK_DIR="" # scratch dir a dry run owns, removed on exit
# Only ever removes the scratch directory this script created for itself; pack
# bytes on disk are retired to .AGENTTRASH/, never deleted (see --force).
cleanup_work_dir() {
if [ -n "$WORK_DIR" ]; then
rm -rf -- "$WORK_DIR"
fi
}
trap cleanup_work_dir EXIT
die() {
printf 'ERROR: %s\n' "$*" >&2
exit 1
}
usage() {
sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'
}
# fcmp A OP B — float comparison (awk); OP is one of < <= > >= == and A/B are
# already validated as plain decimals, so nothing user-shaped reaches awk as
# program text.
fcmp() {
awk -v a="$1" -v b="$3" "BEGIN { exit !(a $2 b) }"
}
# floorn X — greatest integer <= X, printed. Only ever called on positives.
floorn() {
awk -v x="$1" 'BEGIN { printf "%d\n", int(x) }'
}
is_seconds() {
[[ "$1" =~ ^[0-9]+([.][0-9]+)?$ ]]
}
is_hex64() {
[[ "$1" =~ ^[0-9a-f]{64}$ ]]
}
# ---------------------------------------------------------------- arguments --
VOICE=""
PACKS_DIR=""
ARG_SOURCE=""
ARG_OFFSET=""
ARG_DURATION=""
ARG_SHA=""
ARG_TOOL=""
DRY_RUN=0
FORCE=0
BUILD_ALL=0
while [ "$#" -gt 0 ]; do
case "$1" in
--env) ENV_FILE="${2:?--env needs a path}"; shift 2 ;;
--source) ARG_SOURCE="${2:?--source needs a file}"; shift 2 ;;
--offset) ARG_OFFSET="${2:?--offset needs seconds}"; shift 2 ;;
--duration) ARG_DURATION="${2:?--duration needs seconds}"; shift 2 ;;
--sha256) ARG_SHA="${2:?--sha256 needs a hex digest}"; shift 2 ;;
--tool) ARG_TOOL="${2:?--tool needs a path}"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
--force) FORCE=1; shift ;;
--all) BUILD_ALL=1; shift ;;
-h|--help) usage; exit 0 ;;
--*) die "unknown option: $1 (see --help)" ;;
*)
if [ -z "$VOICE" ]; then
VOICE="$1"
elif [ -z "$PACKS_DIR" ]; then
PACKS_DIR="$1"
else
die "unexpected argument: $1"
fi
shift
;;
esac
done
[ -f "$ENV_FILE" ] || die "tunables file not found: $ENV_FILE"
for tool in ffmpeg ffprobe jq sha256sum; do
command -v "$tool" >/dev/null 2>&1 || die "$tool is required and not on PATH"
done
# shellcheck disable=SC1090
source "$ENV_FILE"
# Tunables the env file owes us. Defaults are the engine facts read out of the
# pinned tree; an env file that omits one still builds against the truth.
VOICE_PACKS="${VOICE_PACKS:-}"
SAMPLE_RATE="${VOICE_PACK_SAMPLE_RATE:-24000}"
MIN_SECONDS="${VOICE_PACK_MIN_SECONDS:-18}"
MAX_SECONDS="${VOICE_PACK_MAX_SECONDS:-25}"
FRAME_RATE="${VOICE_PACK_FRAME_RATE:-12.5}"
RVQ_BYTES_PER_FRAME="${VOICE_PACK_RVQ_BYTES_PER_FRAME:-22}"
SPK_BYTES="${VOICE_PACK_SPK_BYTES:-8192}"
RVQ_MIN_BYTES="${VOICE_PACK_RVQ_MIN_BYTES:-5456}"
RVQ_MAX_BYTES="${VOICE_PACK_RVQ_MAX_BYTES:-6864}"
TOOL="${ARG_TOOL:-${VOICE_PACK_TOOL:-}}"
CODEC_MODEL="${VOICE_PACK_CODEC_MODEL:-}"
TALKER_MODEL="${VOICE_PACK_TALKER_MODEL:-}"
ENGINE_REPO="${VOICE_PACK_ENGINE_REPO:-https://github.com/ServeurpersoCom/qwentts.cpp}"
ENGINE_COMMIT="${VOICE_PACK_ENGINE_COMMIT:-5916bf7c79dbb4e66fc059cdc507a79a23a28bb4}"
DEVICE="${VOICE_PACK_DEVICE:-CPU}"
THREADS="${VOICE_PACK_THREADS:-8}"
NICE_LEVEL="${VOICE_PACK_NICE:-19}"
TRASH_DIRNAME="${VOICE_PACK_TRASH_DIRNAME:-.AGENTTRASH}"
DEFAULT_PACKS_DIR="${VOICE_PACK_DIR:-/home/dank/voices}"
[ -n "$PACKS_DIR" ] || PACKS_DIR="$DEFAULT_PACKS_DIR"
[ -n "$PACKS_DIR" ] || die "no packs directory (pass it as an argument)"
is_seconds "$SAMPLE_RATE" || die "VOICE_PACK_SAMPLE_RATE is not a number: $SAMPLE_RATE"
is_seconds "$MIN_SECONDS" || die "VOICE_PACK_MIN_SECONDS is not a number: $MIN_SECONDS"
is_seconds "$MAX_SECONDS" || die "VOICE_PACK_MAX_SECONDS is not a number: $MAX_SECONDS"
# hop / frame arithmetic, derived once: the codec runs at FRAME_RATE Hz over
# SAMPLE_RATE audio, so one frame is HOP_SAMPLES samples and one second is
# FRAME_RATE frames.
HOP_SAMPLES="$(floorn "$(awk -v r="$SAMPLE_RATE" -v f="$FRAME_RATE" 'BEGIN { print r / f }')")"
[ "$HOP_SAMPLES" -gt 0 ] || die "VOICE_PACK_FRAME_RATE ($FRAME_RATE) exceeds the sample rate"
BYTES_PER_SECOND="$(awk -v b="$RVQ_BYTES_PER_FRAME" -v f="$FRAME_RATE" 'BEGIN { print b * f }')"
# rvq_bytes_for DURATION — packed size the extractor will emit for a crop this
# long: whole frames only, because it truncates to the hop boundary.
rvq_bytes_for() {
local samples frames
samples="$(awk -v d="$1" -v r="$SAMPLE_RATE" 'BEGIN { printf "%d\n", int(d * r + 0.5) }')"
frames=$((samples / HOP_SAMPLES))
printf '%d\n' "$((frames * RVQ_BYTES_PER_FRAME))"
}
# ------------------------------------------------------------------- build --
build_voice() {
local voice="$1"
# The pack dir name is the voice name on the wire: app.py's VOICE_NAME_RE
# matches it exactly while the engine lowercases names internally, so an
# uppercase or punctuated name would be reported one way by the façade and
# another by the engine. Lowercase only, and nothing that could escape
# $PACKS_DIR.
[[ "$voice" =~ ^[a-z0-9][a-z0-9._-]{0,63}$ ]] ||
die "voice name '$voice' is not a lowercase [a-z0-9._-] name starting with alphanumerics (app.py resolves it verbatim; the engine lowercases it)"
local prefix
prefix="VOICE_PACK_$(printf '%s' "$voice" | tr '[:lower:]' '[:upper:]' | tr -c '[:alnum:]' '_')"
local source_var="${prefix}_SOURCE" offset_var="${prefix}_OFFSET"
local duration_var="${prefix}_DURATION" sha_var="${prefix}_SHA256"
local source="${ARG_SOURCE:-${!source_var:-}}"
local offset="${ARG_OFFSET:-${!offset_var:-}}"
local duration="${ARG_DURATION:-${!duration_var:-}}"
local want_sha="${ARG_SHA:-${!sha_var:-}}"
if [ -z "$voice" ] || [ -z "$source" ]; then
die "no source for '$voice': pass --source or set ${source_var} in $ENV_FILE"
fi
[ -n "$offset" ] || die "no crop offset for '$voice': pass --offset or set ${offset_var}"
[ -n "$duration" ] || die "no crop duration for '$voice': pass --duration or set ${duration_var}"
is_seconds "$offset" || die "crop offset '$offset' is not a number of seconds"
is_seconds "$duration" || die "crop duration '$duration' is not a number of seconds"
[ -f "$source" ] && [ -r "$source" ] || die "source not readable: $source"
# === THE GATE. Source seconds 0..10 are forbidden voice material, full stop.
if ! fcmp "$offset" '>=' "$SOURCE_FORBIDDEN_SECONDS"; then
die "source-material invariant violated: the crop must start past the first ${SOURCE_FORBIDDEN_SECONDS} s of the source file, and '$voice' asks for ${offset} s into $source. Pick an offset at or past ${SOURCE_FORBIDDEN_SECONDS} s — nothing was written."
fi
local pack_dir="$PACKS_DIR/$voice"
if [ "$DRY_RUN" -ne 1 ] && [ -e "$pack_dir" ] && [ "$FORCE" -ne 1 ]; then
die "$pack_dir already exists; nothing was touched. Re-run with --force to retire it to $PACKS_DIR/$TRASH_DIRNAME/ and rebuild."
fi
local source_duration source_bytes source_sha
source_duration="$(ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 "$source")"
is_seconds "$source_duration" || die "ffprobe reports no numeric duration for $source: '$source_duration'"
source_bytes="$(stat -c%s "$source")"
if ! fcmp "$(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }')" '<=' "$source_duration"; then
die "crop ends at $(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }') s but $source is only $source_duration s; the crop would come out short"
fi
if ! fcmp "$duration" '>=' "$MIN_SECONDS" || ! fcmp "$duration" '<=' "$MAX_SECONDS"; then
die "crop duration $duration s is outside the ${MIN_SECONDS}-${MAX_SECONDS} s reference window"
fi
local crop_end planned_rvq
crop_end="$(awk -v a="$offset" -v d="$duration" 'BEGIN { print a + d }')"
planned_rvq="$(rvq_bytes_for "$duration")"
if ! fcmp "$planned_rvq" '>=' "$RVQ_MIN_BYTES" || ! fcmp "$planned_rvq" '<=' "$RVQ_MAX_BYTES"; then
die "a $duration s crop packs to ${planned_rvq} B of .rvq, outside the recorded engine window ${RVQ_MIN_BYTES}-${RVQ_MAX_BYTES} B; lengthen the crop (that window starts at $(awk -v m="$RVQ_MIN_BYTES" -v b="$BYTES_PER_SECOND" 'BEGIN { print m / b }') s) — the source, not the codec, sets this floor"
fi
echo "VOICE $voice"
echo "SOURCE $source ($source_bytes bytes, ${source_duration} s)"
echo "CROP ${offset} s .. ${crop_end} s (${duration} s) — past the ${SOURCE_FORBIDDEN_SECONDS} s forbidden region"
source_sha="$(sha256sum "$source" | awk '{ print $1 }')"
if [ -n "$want_sha" ]; then
is_hex64 "$want_sha" || die "expected sha256 '$want_sha' is not 64 hex digits"
[ "$source_sha" = "$want_sha" ] ||
die "source sha256 is $source_sha, expected $want_sha: $source is not the file these offsets were cut from"
fi
echo "SHA256 $source_sha"
# Where the crop is cut. A dry run keeps its work in a scratch dir it owns;
# a real build stages INSIDE the pack dir so the final rename is on one
# filesystem and the pack becomes visible only when it is whole.
local stage_dir
if [ "$DRY_RUN" -eq 1 ]; then
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/voice-pack-$voice.XXXXXX")"
stage_dir="$WORK_DIR"
else
mkdir -p "$PACKS_DIR"
if [ -e "$pack_dir" ]; then
local trash
trash="$PACKS_DIR/$TRASH_DIRNAME/$voice-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$(dirname "$trash")"
mv "$pack_dir" "$trash"
echo "RETIRED $pack_dir -> $trash (nothing deleted)"
fi
mkdir "$pack_dir" || die "cannot create $pack_dir (it may exist now); nothing was overwritten"
stage_dir="$pack_dir"
fi
local stage_wav="$stage_dir/staging-reference.wav"
nice -n "$NICE_LEVEL" env OMP_NUM_THREADS="$THREADS" \
ffmpeg -hide_banner -loglevel error -y \
-ss "$offset" -t "$duration" -i "$source" \
-map 0:a:0 -vn -ac 1 -ar "$SAMPLE_RATE" -c:a pcm_s16le "$stage_wav"
local wav_rate wav_channels wav_codec wav_duration
wav_rate="$(ffprobe -v error -select_streams a:0 -show_entries stream=sample_rate -of default=nw=1:nk=1 "$stage_wav")"
wav_channels="$(ffprobe -v error -select_streams a:0 -show_entries stream=channels -of default=nw=1:nk=1 "$stage_wav")"
wav_codec="$(ffprobe -v error -select_streams a:0 -show_entries stream=codec_name -of default=nw=1:nk=1 "$stage_wav")"
wav_duration="$(ffprobe -v error -select_streams a:0 -show_entries stream=duration -of default=nw=1:nk=1 "$stage_wav")"
is_seconds "$wav_rate" || die "ffprobe reports no numeric sample rate for the crop: '$wav_rate'"
is_seconds "$wav_duration" || die "ffprobe reports no numeric duration for the crop: '$wav_duration'"
[ "$wav_rate" = "$SAMPLE_RATE" ] || die "reference.wav came out at ${wav_rate} Hz, not ${SAMPLE_RATE}"
[ "$wav_channels" = "1" ] || die "reference.wav came out with $wav_channels channels, not mono"
[ "$wav_codec" = "pcm_s16le" ] || die "reference.wav came out as $wav_codec, not the pcm_s16le the law names"
if ! fcmp "$wav_duration" '>=' "$MIN_SECONDS" || ! fcmp "$wav_duration" '<=' "$MAX_SECONDS"; then
die "reference.wav is ${wav_duration} s, outside the ${MIN_SECONDS}-${MAX_SECONDS} s window"
fi
echo "REFERENCE ${wav_rate} Hz ${wav_codec} mono, ${wav_duration} s, $(stat -c%s "$stage_wav") bytes"
local spk_file="$stage_dir/staging-reference.spk" rvq_file="$stage_dir/staging-reference.rvq"
local stage_log="$stage_dir/staging-qwen-codec.log"
local spk_bytes=0 rvq_bytes=0 produced=false
if [ "$DRY_RUN" -eq 1 ]; then
if [ -n "$TOOL" ] && [ -x "$TOOL" ]; then
echo "EXTRACTOR $TOOL (present; --dry-run will not run it)"
else
echo "EXTRACTOR not present (${TOOL:-unset}) — --dry-run does not need it"
fi
echo "PLANNED reference.spk ${SPK_BYTES} B, reference.rvq $(rvq_bytes_for "$wav_duration") B"
else
[ -n "$TOOL" ] && [ -x "$TOOL" ] ||
die "speaker extractor missing or not executable: ${TOOL:-VOICE_PACK_TOOL unset} (use --dry-run to build everything up to this point)"
[ -f "$CODEC_MODEL" ] || die "codec model missing: $CODEC_MODEL"
[ -f "$TALKER_MODEL" ] || die "talker model missing: $TALKER_MODEL"
# The extractor takes the WAV and writes its outputs next to it with the
# extension swapped — there is no -o, so the staged name decides where
# the artifacts land.
local extractor=("$TOOL" --model "$CODEC_MODEL" --talker "$TALKER_MODEL" -i "$stage_wav")
if [ -n "$DEVICE" ]; then
extractor+=(--device "$DEVICE")
fi
echo "EXTRACT $(basename "$TOOL") at $ENGINE_COMMIT (device ${DEVICE:-auto}, $THREADS threads)"
if ! nice -n "$NICE_LEVEL" env OMP_NUM_THREADS="$THREADS" \
"${extractor[@]}" 2>&1 | tee "$stage_log" >&2; then
die "$TOOL failed on $stage_wav (its output is in $stage_log); the pack was not published"
fi
[ -f "$spk_file" ] || die "$TOOL wrote no $spk_file (see $stage_log)"
[ -f "$rvq_file" ] || die "$TOOL wrote no $rvq_file (see $stage_log)"
spk_bytes="$(stat -c%s "$spk_file")"
rvq_bytes="$(stat -c%s "$rvq_file")"
local expect_rvq
expect_rvq="$(rvq_bytes_for "$wav_duration")"
[ "$spk_bytes" = "$SPK_BYTES" ] ||
die "reference.spk is $spk_bytes B, expected ${SPK_BYTES} B (enc_dim x f32). The engine changed its speaker encoder — see [[Voice Packs]] before shipping this pack"
[ "$rvq_bytes" = "$expect_rvq" ] ||
die "reference.rvq is $rvq_bytes B, expected ${expect_rvq} B for a ${wav_duration} s crop (22 B per 12.5 Hz frame)"
if ! fcmp "$rvq_bytes" '>=' "$RVQ_MIN_BYTES" || ! fcmp "$rvq_bytes" '<=' "$RVQ_MAX_BYTES"; then
die "reference.rvq is $rvq_bytes B, outside the recorded engine window ${RVQ_MIN_BYTES}-${RVQ_MAX_BYTES} B — an engine bump? see [[Voice Packs]]"
fi
produced=true
echo "ARTIFACTS reference.spk ${spk_bytes} B, reference.rvq ${rvq_bytes} B"
fi
local built stage_manifest
built="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
stage_manifest="$stage_dir/staging-manifest.json"
jq -n \
--arg voice "$voice" \
--arg built "$built" \
--arg builder "scripts/build-voice-pack.sh" \
--arg env_file "$ENV_FILE" \
--arg src "$source" \
--arg src_sha "$source_sha" \
--argjson src_bytes "$source_bytes" \
--argjson src_duration "$source_duration" \
--argjson offset "$offset" \
--argjson end "$crop_end" \
--argjson duration "$duration" \
--argjson forbidden "$SOURCE_FORBIDDEN_SECONDS" \
--argjson rate "$SAMPLE_RATE" \
--argjson wav_duration "$wav_duration" \
--arg wav_codec "$wav_codec" \
--argjson wav_channels "$wav_channels" \
--argjson wav_bytes "$(stat -c%s "$stage_wav")" \
--arg tool "${TOOL:-unset}" \
--arg repo "$ENGINE_REPO" \
--arg commit "$ENGINE_COMMIT" \
--arg codec "$CODEC_MODEL" \
--arg talker "$TALKER_MODEL" \
--arg device "$DEVICE" \
--argjson produced "$produced" \
--argjson spk_bytes "$spk_bytes" \
--argjson rvq_bytes "$rvq_bytes" \
--argjson planned_rvq "$(rvq_bytes_for "$wav_duration")" \
--argjson dry_run "$DRY_RUN" \
'{
voice: $voice,
kind: "voice-pack",
built_utc: $built,
builder: $builder,
tunables: $env_file,
dry_run: ($dry_run == 1),
source: {
path: $src,
sha256: $src_sha,
bytes: $src_bytes,
duration_seconds: $src_duration
},
crop: {
offset_seconds: $offset,
end_seconds: $end,
duration_seconds: $duration
},
reference: {
file: "reference.wav",
codec: $wav_codec,
sample_rate: $rate,
channels: $wav_channels,
duration_seconds: $wav_duration,
bytes: $wav_bytes
},
artifacts: {
produced: $produced,
tool: $tool,
engine_repo: $repo,
engine_commit: $commit,
device: $device,
codec_model: $codec,
talker_model: $talker,
spk: {file: "reference.spk", bytes: $spk_bytes},
rvq: {file: "reference.rvq", bytes: $rvq_bytes, planned_bytes: $planned_rvq}
},
invariant: {
name: "source-material invariant",
rule: "a voice crop MUST start past the first 10 seconds of the source file",
forbidden_source_seconds: $forbidden,
crop_offset_seconds: $offset
}
}' >"$stage_manifest"
# Assert the gate a second time, FROM the manifest, next to the bytes it
# describes. Anything short of a pass leaves the staged pack unpublished.
local recorded_offset recorded_sha
recorded_offset="$(jq -r '.crop.offset_seconds' "$stage_manifest")"
recorded_sha="$(jq -r '.source.sha256' "$stage_manifest")"
jq -e --argjson min "$SOURCE_FORBIDDEN_SECONDS" \
'(.crop.offset_seconds | type == "number") and .invariant.forbidden_source_seconds == $min and (.crop.offset_seconds >= $min) and (.invariant.name == "source-material invariant")' \
"$stage_manifest" >/dev/null ||
die "manifest assertion failed: .crop.offset_seconds ($recorded_offset) does not clear the ${SOURCE_FORBIDDEN_SECONDS} s source-material invariant"
[ "$recorded_sha" = "$source_sha" ] ||
die "manifest assertion failed: .source.sha256 ($recorded_sha) is not the hash of $source ($source_sha)"
jq -e '(.voice | type == "string") and (.reference.sample_rate | type == "number") and (.reference.duration_seconds | type == "number") and (.artifacts.engine_commit | length == 40)' \
"$stage_manifest" >/dev/null ||
die "manifest assertion failed: voice / reference / engine provenance fields are not all present"
echo "MANIFEST crop offset $recorded_offset s clears the ${SOURCE_FORBIDDEN_SECONDS} s invariant"
if [ "$DRY_RUN" -eq 1 ]; then
echo "MANIFEST_BEGIN"
cat "$stage_manifest"
echo "MANIFEST_END"
echo "PLAN would write $pack_dir/{reference.wav,reference.spk,reference.rvq,manifest.json}"
echo "DRYRUN $voice: inputs valid, crop cut and probed, extractor not run"
else
# reference.wav is the gate the shim reads, so it lands last: the voice
# name never exists on disk in a state the artifacts are not behind it.
mv "$stage_log" "$pack_dir/qwen-codec.log"
mv "$spk_file" "$pack_dir/reference.spk"
mv "$rvq_file" "$pack_dir/reference.rvq"
mv "$stage_manifest" "$pack_dir/manifest.json"
mv "$stage_wav" "$pack_dir/reference.wav"
echo "BUILT $pack_dir (voice '$voice': ${wav_duration} s, spk ${spk_bytes} B, rvq ${rvq_bytes} B)"
echo "VOICE_PACK_BUILT $voice $pack_dir"
fi
}
if [ "$BUILD_ALL" -eq 1 ]; then
[ -z "$VOICE" ] || die "--all takes no voice name (got '$VOICE'); the list comes from VOICE_PACKS"
for flag in "$ARG_SOURCE" "$ARG_OFFSET" "$ARG_DURATION" "$ARG_SHA"; do
[ -z "$flag" ] || die "--all reads every crop from $ENV_FILE; --source/--offset/--duration/--sha256 would apply one value to all of them"
done
read -r -a voices <<<"$VOICE_PACKS"
[ "${#voices[@]}" -gt 0 ] || die "VOICE_PACKS is empty in $ENV_FILE"
for voice in "${voices[@]}"; do
build_voice "$voice"
done
else
[ -n "$VOICE" ] || { usage >&2; die "no voice name given"; }
build_voice "$VOICE"
fi
+16
View File
@@ -26,6 +26,7 @@ SERVICE_ORDER = (
"marreta", "marreta",
"pihole", "pihole",
"open-webui", "open-webui",
"open-notebook",
"gitea-postgres", "gitea-postgres",
"gitea-server", "gitea-server",
"comfyui", "comfyui",
@@ -74,6 +75,7 @@ DANKCLAW_SERVICES = {
"marreta", "marreta",
"pihole", "pihole",
"open-webui", "open-webui",
"open-notebook",
"gitea-postgres", "gitea-postgres",
"gitea-server", "gitea-server",
"comfyui", "comfyui",
@@ -316,6 +318,14 @@ def services_for_path(path: str, target: str) -> set[str]:
if len(parts) >= 2 and parts[1] == "open-webui": if len(parts) >= 2 and parts[1] == "open-webui":
_add(services, "open-webui") _add(services, "open-webui")
return services return services
if len(parts) >= 2 and parts[1] == "open-notebook":
# One deployable owns the app AND its SurrealDB, like the marreta
# app/companion pair: the app alone cannot write a note and the
# database alone is a volume nothing reads. Any change under the
# directory (unit, network, or the interim episode mount) therefore
# plans the pair, which deploy_open_notebook starts database-first.
_add(services, "open-notebook")
return services
# frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host # frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
# build, root unit): its files live in scripts/ and map there. # build, root unit): its files live in scripts/ and map there.
if len(parts) >= 2 and parts[1] == "observability": if len(parts) >= 2 and parts[1] == "observability":
@@ -426,6 +436,12 @@ def services_for_path(path: str, target: str) -> set[str]:
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches # serving Qwen3.8-Flash-Next Q8_0. The download script fetches
# the payload, which is host-side state, so it selects nothing. # the payload, which is host-side state, so it selects nothing.
_add(services, "frank-archon") _add(services, "frank-archon")
if name == "open-notebook.env":
# open-notebook's tunables live here — the only repo-tracked part of
# either container's environment (the credentials are seeded on the
# host, so they are never a deploy trigger). Changing it is a
# restart of the pair, which the single deployable performs.
_add(services, "open-notebook")
# Deployment/test/benchmark scripts and other repo tooling do not # Deployment/test/benchmark scripts and other repo tooling do not
# restart a runtime service. # restart a runtime service.
return services return services
+99
View File
@@ -90,6 +90,7 @@ service_is_allowed() {
dankclaw:authentik-postgres|dankclaw:authentik-server|\ dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\ dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\ dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
dankclaw:open-notebook|\
dankclaw:frank-dragoon|dankclaw:frank-zealot|\ dankclaw:frank-dragoon|dankclaw:frank-zealot|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\ dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\ dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
@@ -646,6 +647,103 @@ deploy_open_webui() {
systemctl --user restart open-webui.service systemctl --user restart open-webui.service
} }
# open-notebook is the fleet's podcast generator: one app container (API,
# worker and web UI under supervisord) in front of a SurrealDB container, with
# a private bridge between them. One handler owns the pair, because restarting
# the app without its database yields a UI that answers /health and fails every
# write; scripts/deploy-plan.py therefore maps every path of the pair here.
#
# Credentials follow the marreta convention (per-service file under the deploy
# user's home, mode 600, seeded from dankclaw/.env) with one difference: they
# are not optional. An empty OPEN_NOTEBOOK_PASSWORD is "authentication OFF",
# not "unconfigured" — the API, episode audio included, then answers anyone on
# the LAN — and an empty SURREAL_PASS lets the database come up with different
# root credentials than the app will present. So this handler refuses to start
# either unit until every key carries a value. An existing file is NEVER
# rewritten: regenerating an operator-set password would desync it from the
# credentials already stored in the database.
seed_open_notebook_env() {
local dest=/home/dank/Containers/open-notebook/env
local var missing=0
for var in OPEN_NOTEBOOK_ENCRYPTION_KEY OPEN_NOTEBOOK_PASSWORD SURREAL_PASSWORD SURREAL_PASS; do
grep -q "^${var}=." "$dest" 2>/dev/null || missing=1
done
if [[ "$missing" == 0 ]]; then
return 0
fi
if [[ ! -e "$dest" ]]; then
if [[ ! -f dankclaw/.env ]]; then
echo "open-notebook: neither $dest nor dankclaw/.env exists; put OPEN_NOTEBOOK_ENCRYPTION_KEY, OPEN_NOTEBOOK_PASSWORD and SURREAL_PASSWORD in dankclaw/.env (see dankclaw/.env.example)" >&2
return 1
fi
mkdir -p /home/dank/Containers/open-notebook
{
awk -F= '/^OPEN_NOTEBOOK_ENCRYPTION_KEY=/ || /^OPEN_NOTEBOOK_PASSWORD=/ || /^SURREAL_PASSWORD=/ { print }' \
dankclaw/.env
# The `surreal` CLI reads SURREAL_PASS from the environment while
# the app reads SURREAL_PASSWORD. One value, written twice, so the
# pair can never be seeded with two different root passwords.
sed -n 's/^SURREAL_PASSWORD=/SURREAL_PASS=/p' dankclaw/.env
} > "$dest"
chmod 600 "$dest"
fi
for var in OPEN_NOTEBOOK_ENCRYPTION_KEY OPEN_NOTEBOOK_PASSWORD SURREAL_PASSWORD SURREAL_PASS; do
grep -q "^${var}=." "$dest" \
|| { echo "open-notebook: $dest carries no value for ${var}; fill it in directly (or in dankclaw/.env before the first seed) — a blank value here means an unauthenticated service or a mismatched database password" >&2; return 1; }
done
}
# Both units ship Pull=never: the digest pins in the .container files are the
# contract, and the generator must never re-resolve a moving tag at start. That
# makes fetching a pin a precondition the deployer satisfies once per pin
# change; the existence check keeps every later re-deploy offline-safe.
ensure_pinned_image() {
local image="$1"
if podman image exists "$image"; then
echo "open-notebook: $image is already in local storage"
return 0
fi
echo "open-notebook: pulling pinned image $image"
podman pull "$image"
}
deploy_open_notebook() {
seed_open_notebook_env || return 1
ensure_pinned_image 'docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723' || return 1
ensure_pinned_image 'docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212' || return 1
# install_quadlet copies the sibling open-notebook.network with the app
# unit. That cp is load-bearing, not tidiness: when a referenced Network=
# file is absent the v1 generator refuses to produce the container unit at
# all ("requested Quadlet unit open-notebook.network was not found").
install_quadlet open-notebook-surrealdb dankclaw/open-notebook/quadlet open-notebook-surrealdb
install_quadlet open-notebook dankclaw/open-notebook/quadlet open-notebook
systemctl --user start open-notebook-network.service
# Interim episode destination (Gitea issue #7, `human-only`): finished
# episode audio is bind-mounted out to this host directory. Created here as
# the deploy user so the container's mapped root writes files dank can
# read, and so the dependency is explicit rather than podman's default.
mkdir -p /home/dank/media/podcasts
# No `systemctl --user enable`: generated Quadlet units cannot be enabled
# on this host (see deploy_authentik); boot persistence is the [Install]
# section inside each .container file. Database first, and both on every
# run — a stale app left attached to a recreated database is exactly the
# half-deployed state this handler exists to prevent.
systemctl --user restart open-notebook-surrealdb.service
wait_container_healthy open-notebook-surrealdb 90 || return 1
systemctl --user restart open-notebook.service
# The gate is the API's own health route: upstream serves `GET /health` on
# 5055 with {"status":"healthy"} and exempts it from the password
# middleware, so it stays a real answer once OPEN_NOTEBOOK_PASSWORD is set.
# Host port 5056, because seerr already holds 127.0.0.1:5055. The UI on
# 8502 is deliberately not gated: supervisord starts it only after the API
# answers (the image's wait-for-api.sh), so this gate already implies it.
wait_for_health open-notebook http://127.0.0.1:5056/health 180
}
deploy_comfyui() { deploy_comfyui() {
# Build first: the unit's Image= resolves against local storage, so the # Build first: the unit's Image= resolves against local storage, so the
# image has to exist before the generator starts a container from it. # image has to exist before the generator starts a container from it.
@@ -1573,6 +1671,7 @@ deploy_service() {
gluetun) deploy_gluetun ;; gluetun) deploy_gluetun ;;
searxng-valkey) deploy_searxng_valkey ;; searxng-valkey) deploy_searxng_valkey ;;
open-webui) deploy_open_webui ;; open-webui) deploy_open_webui ;;
open-notebook) deploy_open_notebook ;;
pihole) deploy_pihole ;; pihole) deploy_pihole ;;
gitea-postgres) deploy_gitea_unit gitea-postgres gitea-postgres ;; gitea-postgres) deploy_gitea_unit gitea-postgres gitea-postgres ;;
gitea-server) deploy_gitea_unit gitea-server gitea-server ;; gitea-server) deploy_gitea_unit gitea-server gitea-server ;;
+1
View File
@@ -141,6 +141,7 @@ valid_service() {
dankclaw:alertmanager|\ dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\ dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\ dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:open-notebook|\
dankclaw:frank-dragoon|dankclaw:frank-zealot|\ dankclaw:frank-dragoon|dankclaw:frank-zealot|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\ dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\ dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
+100
View File
@@ -0,0 +1,100 @@
# open-notebook.env — configuration for **open-notebook**, the fleet's podcast
# generator on Dankclaw (UI host port **8502**, API host port **5056**). Read by
# BOTH Quadlet units (dankclaw/open-notebook/quadlet/) as the FIRST
# EnvironmentFile= line, which is why every path and host name below is the
# value the CONTAINER must see.
#
# TWO FILES, ONE ORDER. This file is tracked; it holds knobs, not credentials.
# The three secret keys below are declared EMPTY on purpose — the `scripts/*.env`
# gitignore exception exists for "paths and numbers, never credentials" — and the
# real values live in the host file the deployer seeds at
# /home/dank/Containers/open-notebook/env (mode 600, seeded from dankclaw/.env,
# the same convention grafana/marreta/pihole follow). Both files are handed to
# podman in EnvironmentFile= order and the LAST one wins (verified against the
# generated run line), so the host file overrides these blanks. Do not reorder
# the two lines.
#
# WHY THE DEPLOYER REFUSES TO START THESE UNITS WITHOUT THE SECRET FILE. An empty
# OPEN_NOTEBOOK_PASSWORD is not "unconfigured", it is **authentication off**, and
# the episode-audio download is one of the routes the API serves without a
# session; an empty SURREAL_PASS lets the database come up with different root
# credentials than the app will present; an empty OPEN_NOTEBOOK_ENCRYPTION_KEY
# means no provider credential can be saved at all. So the guard is in
# deploy-remote.sh (seed_open_notebook_env), not in the app's opinion.
#
# THE CPU-SHUTDOWN GUARD (the two knobs that matter most on this box). Defaults
# are TTS_BATCH_SIZE=5 and OPEN_NOTEBOOK_WORKER_MAX_TASKS=5 — five parallel
# synthesis requests from five parallel jobs. frank-tts answers each request by
# spawning its own engine process (~8 GiB resident, 8 threads, measured in the
# frank-tts CPU trial of 2026-10-18), so the defaults mean ~40 GiB and forty
# threads aimed at a machine that also serves frank-dragoon and frank-zealot and
# whose TTS engine runs on the SAME CPU. Both numbers are 1 here: one episode at
# a time, one voice line synthesized at a time. That is a throughput choice, not
# a correctness one — a podcast is a background job — and it is the difference
# between a slow render and a box that stops answering the text endpoints
# mid-sentence. Raise them only with a TTS engine that batches internally.
#
# THE TTS CLOCK. ESPERANTO_TTS_TIMEOUT is the worker's patience per DIALOGUE
# LINE, not per episode: one episode is dozens of POSTs to /v1/audio/speech
# (the app synthesises each speaker turn separately and concatenates the clips
# with ffmpeg afterwards), and a CPU engine takes single-digit seconds for a
# line, with the tail of a long turn well past the 300 s default. 660 sits one
# round-trip OUTSIDE SHIM_TIMEOUT=600 in scripts/frank-tts.env — the shim's own
# per-request ceiling — so a frank-tts give-up arrives as frank-tts's error and
# not as open-notebook abandoning a synthesis still in flight. An abandoned
# request does not cancel the engine process; it just leaves the episode stuck in
# "generating" while the audio arrives.
#
# THE SCRIPTWRITER. ESPERANTO_LLM_TIMEOUT is the app's own default (180 s), kept
# visible beside the TTS number because both belong to one podcast run: the
# outline call and the transcript call are the slow text work, and frank-zealot
# answers them well inside 180. Provider choice is made in the UI, not here —
# see [[Open Notebook]] for the one-time credential provisioning — and it is
# frank-zealot on purpose: script writing asks for a JSON-schema response, which
# the OpenAI-compatible llama.cpp endpoint is known to serve, while halogen's
# structured-output behaviour on frank-dragoon is UNVERIFIED. halogen remains
# the unproven alternative, not the fallback.
#
# THE DATABASE COORDINATES. SURREAL_URL addresses the sibling container BY NAME
# on systemd-open-notebook (container DNS is scoped to the owning network, so
# this is the only form that works). NAMESPACE/DATABASE/USER are plain names.
# SURREAL_PASSWORD and SURREAL_PASS are ONE value under two spellings: the app
# reads SURREAL_PASSWORD, the `surreal` CLI reads SURREAL_PASS. Keep them equal;
# the deployer writes both into the seeded file from a single SURREAL_PASSWORD in
# dankclaw/.env, so hand-editing one line is the only way to desync them.
SURREAL_URL=ws://open-notebook-surrealdb:8000/rpc
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=open_notebook
SURREAL_USER=open_notebook
# Secrets — filled on the host, never here (see the header).
SURREAL_PASSWORD=
SURREAL_PASS=
OPEN_NOTEBOOK_ENCRYPTION_KEY=
OPEN_NOTEBOOK_PASSWORD=
# WHERE THE BROWSER FINDS THE API. The UI auto-detects `<host-you-opened>:5055`,
# and 5055 on this host belongs to seerr, so the API leaves the container on 5056
# and this line has to say so. Both origins are listed in CORS_ORIGINS because
# the API is called cross-origin from the UI's own port: the LAN origin is the
# normal one, the localhost one covers a browser opened on Dankclaw itself. The
# app logs a warning at start while CORS_ORIGINS is unset and defaults to `*`,
# which is a wildcard in front of an endpoint that serves every episode's audio.
API_URL=http://192.168.0.143:5056
CORS_ORIGINS=http://192.168.0.143:8502,http://localhost:8502
# Concurrency: see "THE CPU-SHUTDOWN GUARD" above. Defaults are 5 and 5.
TTS_BATCH_SIZE=1
OPEN_NOTEBOOK_WORKER_MAX_TASKS=1
# Timeouts: see "THE TTS CLOCK" and "THE SCRIPTWRITER" above.
ESPERANTO_TTS_TIMEOUT=660
ESPERANTO_LLM_TIMEOUT=180
# Heavy extraction runtimes stay OFF, which is also what keeps boot fast: each
# one installs a multi-gigabyte ML/Chromium stack into the image's venv on first
# startup (cached on the /app/data volume after that). Podcast generation reads
# notes and sources that are already text, so neither is needed here; enable one
# deliberately, per the upstream docs, if source ingestion starts to matter.
# OPEN_NOTEBOOK_ENABLE_DOCLING=true
# OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true
+18
View File
@@ -128,6 +128,16 @@ for path in "${CHANGED[@]}"; do
add_test tests/docs_completeness.bats add_test tests/docs_completeness.bats
add_test tests/deployment_scoping.bats add_test tests/deployment_scoping.bats
;; ;;
# open-notebook is the fleet's podcast generator: an app quadlet, the
# SurrealDB quadlet behind it, the private network between them, and the
# one env file both units read. Like the caddy block above, this case
# has to precede the dankclaw/*.container glob, whose `*` crosses `/`
# and would otherwise swallow dankclaw/open-notebook/quadlet/*.
dankclaw/open-notebook/*|scripts/open-notebook.env)
add_test tests/open_notebook.bats
add_test tests/docs_completeness.bats
add_test tests/deployment_scoping.bats
;;
dankclaw/*.container|dankclaw/*.network) dankclaw/*.container|dankclaw/*.network)
add_test tests/open_webui_quadlet.bats add_test tests/open_webui_quadlet.bats
add_test tests/pihole.bats add_test tests/pihole.bats
@@ -219,6 +229,14 @@ for path in "${CHANGED[@]}"; do
add_test tests/scripts_syntax.bats add_test tests/scripts_syntax.bats
add_test tests/deployment_scoping.bats add_test tests/deployment_scoping.bats
;; ;;
# The voice-pack builder is the only thing that enforces the 10-second
# source-material invariant and the artifact sizes the packs are pinned
# to, so it selects its own seam rather than landing on the generic
# scripts/* syntax fallback. This case has to precede that fallback.
scripts/build-voice-pack.sh|scripts/voice-packs.env)
add_test tests/voice_pack_unit.bats
add_test tests/scripts_syntax.bats
;;
scripts/*) scripts/*)
add_test tests/scripts_syntax.bats add_test tests/scripts_syntax.bats
;; ;;
+123
View File
@@ -0,0 +1,123 @@
# voice-packs.env — tunables for the **frank-tts voice packs**: the three
# numbers per voice (source file, crop offset, crop length) plus the pinned
# engine facts that turn a crop into a pack. Read by
# `scripts/build-voice-pack.sh`, which is the only thing that consumes this
# file — the running service reads none of it, because the packs themselves are
# the interface (see scripts/frank-tts.env for what the shim resolves).
#
# WHAT IS TRACKED HERE AND WHAT IS NOT. Paths and numbers, never credentials,
# which is the same rule the other `scripts/*.env` files follow. The pack
# DIRECTORIES are host state under $VOICE_PACK_DIR and stay out of the repo; the
# crops that produce them are exactly the kind of decision worth a diff, so the
# offsets live here. If a crop changes, change it here and rebuild with
# --force — the manifest inside the pack records what was actually cut.
#
# THE OFFSETS ARE NOT ARBITRARY. Two rules pick them:
# 1. The source-material invariant: seconds 0-10 of any sample file are
# forbidden voice material. The builder refuses a crop that starts inside
# that region and asserts the same thing again from the manifest it just
# wrote, so the values below are checked, not trusted.
# 2. Speech density: the trial cut these from a silencedetect scan of the
# source (noise -35 dB, 0.6 s) and took the longest run of continuous
# speech at or past the scan window — a reference that starts mid-word or
# contains a page-turn of silence clones the artefacts, not the voice.
# serverAppConfigs-docs/Dankclaw/Voice Packs.md carries the recipe.
#
# WHY THE SHA256 LINES EXIST. A crop offset is a statement about one exact
# file: 98.739 s into a re-encode of the same audiobook is a different sentence
# and therefore a different voice. The builder hashes the source, compares it
# with the pinned digest when one is set, and writes the digest into the pack's
# manifest — so a pack can always be traced back to the bytes it was cut from.
# The three values below are the trial's digests (2026-10-18), verified against
# the files on disk.
#
# THE ENGINE BLOCK IS A PIN, NOT A DEFAULT. The speaker artifacts come from
# `qwen-codec` in the qwentts.cpp tree at $VOICE_PACK_ENGINE_COMMIT: it takes a
# 24 kHz WAV, writes the RVQ code stream next to it with the extension swapped,
# and with --talker also runs that talker's speaker encoder to write the
# x-vector .spk. The talker has to be a **Base** build — the customvoice and
# voicedesign GGUFs carry no speaker encoder and the extractor says so. The
# sizes the builder then verifies (8,192 B of .spk, 5,456-6,864 B of .rvq) are
# derived from the codec config in the GGUF, so an engine bump that changes them
# fails the build on purpose; see [[Voice Packs]] for the bump procedure.
#
# HOST PATHS, NOT CONTAINER PATHS. This runs on the host that holds the engine
# build (Dankclaw), outside any container, with no GPU and no fleet endpoint.
# The model paths below are where scripts/download-frank-tts-engine-qwentts.sh
# puts the GGUFs; until that fetch has run, the same two files sit in the trial
# tree at /home/dank/voice-audition/qwentts/models/ and the tool is only in its
# build directory. The SOURCE paths are the third thing to place: the three
# sample files live on the workstation today and have to be copied to
# $VOICE_PACK_SOURCE_DIR on Dankclaw before a build can read them.
# The packs root on the host: the directory frank-tts mounts read-only at
# /voices, one sub-directory per voice.
VOICE_PACK_DIR=/home/dank/voices
# The pack list. `--all` walks it in order; every name has to carry the four
# per-voice values below.
VOICE_PACKS="narrator1 narrator2 narrator3"
# Where the source recordings live on the engine host.
VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources
# narrator1 — Randall Munroe "How To", read by Wil Wheaton. Longest speech run
# at or past the scan window: 98.739-118.636 s, 19.897 s, which packs to the
# 5,456 B floor of the .rvq window.
VOICE_PACK_NARRATOR1_SOURCE=${VOICE_PACK_SOURCE_DIR}/audiosample.mp3
VOICE_PACK_NARRATOR1_OFFSET=98.739
VOICE_PACK_NARRATOR1_DURATION=19.897
VOICE_PACK_NARRATOR1_SHA256=2299510467eb9143455d8c1e6a21601c3b451dd59c1dc99b99d4aacb54d1d850
# narrator2 — "Abyssal Archive" (Lokey). 25 s is the ceiling of the reference
# window and packs to exactly 6,864 B of .rvq.
VOICE_PACK_NARRATOR2_SOURCE=${VOICE_PACK_SOURCE_DIR}/audiosample2.m4b
VOICE_PACK_NARRATOR2_OFFSET=191.13
VOICE_PACK_NARRATOR2_DURATION=25
VOICE_PACK_NARRATOR2_SHA256=8e1e33317baf67b2cca67e4ce1b095ae477a18285c7373ec18ad013f2cc320e0
# narrator3 — "Children of Time" (Adrian Tchaikovsky, author-read).
VOICE_PACK_NARRATOR3_SOURCE=${VOICE_PACK_SOURCE_DIR}/audiosample3.m4b
VOICE_PACK_NARRATOR3_OFFSET=489.232
VOICE_PACK_NARRATOR3_DURATION=25
VOICE_PACK_NARRATOR3_SHA256=6875d2c1db421ab96ccbb68855fa9efd4e054297e94dd1b4345eb6a9869da945
# The pinned extractor and the two GGUFs it needs. $VOICE_PACK_TOOL is a build
# output, not a download: rebuild it at $VOICE_PACK_ENGINE_COMMIT after a bump
# rather than pointing this line at a newer tree.
VOICE_PACK_TOOL=/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec
VOICE_PACK_CODEC_MODEL=/home/dank/models/Qwen3-TTS-GGUF/qwen-tokenizer-12hz-Q8_0.gguf
VOICE_PACK_TALKER_MODEL=/home/dank/models/Qwen3-TTS-GGUF/qwen-talker-1.7b-base-Q4_K_M.gguf
VOICE_PACK_ENGINE_REPO=https://github.com/ServeurpersoCom/qwentts.cpp
VOICE_PACK_ENGINE_COMMIT=5916bf7c79dbb4e66fc059cdc507a79a23a28bb4
# CPU only, and never in the way of the text endpoints on this box: ggml thread
# count and `nice -n`. The extractor picks the best backend it is given, so the
# device is stated rather than guessed.
VOICE_PACK_DEVICE=CPU
VOICE_PACK_THREADS=8
VOICE_PACK_NICE=19
# The reference window and the shape the packs are cut to. 24 kHz mono PCM is
# the codec's own input rate, so no resampling happens inside the engine.
VOICE_PACK_SAMPLE_RATE=24000
VOICE_PACK_MIN_SECONDS=18
VOICE_PACK_MAX_SECONDS=25
# Artifact arithmetic, all four values read out of the pinned tree: the codec
# runs at 12.5 Hz (1,920 samples per frame at 24 kHz), packs 16 codebooks at 11
# bits = 22 bytes per frame, and the Base talker's speaker encoder emits
# enc_dim 2048 f32 = 8,192 bytes. The .rvq window is 22 x 248 = 5,456 B up to
# 22 x 312 = 6,864 B, i.e. crops of 19.84-25 s: an 18 s crop is a legal
# reference.wav but too little code for the engine, and the builder says so
# before it runs anything.
VOICE_PACK_FRAME_RATE=12.5
VOICE_PACK_RVQ_BYTES_PER_FRAME=22
VOICE_PACK_SPK_BYTES=8192
VOICE_PACK_RVQ_MIN_BYTES=5456
VOICE_PACK_RVQ_MAX_BYTES=6864
# Where a --force rebuild puts the pack it replaced. Nothing is ever deleted;
# the retired directory stays readable beside its replacement until a human
# clears it.
VOICE_PACK_TRASH_DIRNAME=.AGENTTRASH
@@ -0,0 +1,132 @@
---
type: reference
description: "open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out — script written by frank-zealot, voices rendered by frank-tts, no cloud provider anywhere in the path."
tags:
- dankclaw
- open-notebook
- podcasts
- quadlet
timestamp: 2026-10-18
---
# Open Notebook
open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-zealot**, the voices are rendered by **frank-tts**, and nothing leaves the LAN.
**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Deploying it is a deliberate next step, not a pending one.
**Host:** Dankclaw (rootless Podman, user `dank`) — **Ports:** 8502 UI and 5056 API, published on the LAN, **no vhost and no Caddy route** — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook
## Facts (verified 2026-10-18 against the pinned images, upstream source, and the live host)
| Item | Value |
|------|-------|
| App image | `docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212` (`open-notebook.container:79`) — the tag upstream's own `docker-compose.yml:27` pins for the multi-container shape. `v1-latest` is a moving tag, so the manifest-list digest (resolved from the registry 2026-10-18) is what makes this reproducible |
| Database image | `docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723` (`open-notebook-surrealdb.container:66`) — that digest is **SurrealDB 2.7.0** (image built 2026-09-23; measured by running the digest: `surreal version`) |
| Do NOT use | the `-single` tags (`v1-latest-single`). That variant bundles SurrealDB inside the app image and is deprecated upstream; here it would give the host a second, hidden database |
| Ports | container 8502 (Next.js UI) + 5055 (FastAPI API) → published `0.0.0.0:8502:8502` and `0.0.0.0:5056:5055` (`open-notebook.container:83-84`). **Host 5055 is taken**: seerr publishes `127.0.0.1:5055` (`dankclaw/seerr/quadlet/seerr.container`, and `ss -ltn` on Dankclaw shows `127.0.0.1:5055` live, 2026-10-18). The API therefore leaves on 5056, which is what forces `API_URL` (below) |
| Database port | 8000 inside its own namespace, **never published** (`open-notebook-surrealdb.container` has no `PublishPort`). Upstream publishes `127.0.0.1:8000` for debugging (`docker-compose.yml:18`); this unit does not, and the seam asserts that it stays that way |
| Data | `open-notebook-data:/app/data` (named volume: uploads, LangGraph SQLite checkpoints, extraction caches) **plus** the interim host bind `/home/dank/media/podcasts:/app/data/podcasts` (`open-notebook.container:87-88`); database in the `open-notebook-db` named volume |
| Networks | app: `dankclaw_default` first, then the private `systemd-open-notebook` (10.89.4.0/24, `open-notebook.network:33`); database: the private bridge only. Order is load-bearing — the generated `--network` flags keep file order, so the default route rides the network where `host.containers.internal` is proven |
| Units | `dankclaw/open-notebook/quadlet/open-notebook.container` → `open-notebook.service`; `open-notebook-surrealdb.container` → `open-notebook-surrealdb.service`; both `Restart=always`, `RestartSec=10` / `5`, `[Install] WantedBy=default.target` — boot-persistent per [[Quadlet Lifecycle]] (generated units cannot be `enable`d on this host; the `[Install]` section is what the v1 generator honors at user-manager start) |
| Health | app: `curl -fsS http://127.0.0.1:5055/health` (`open-notebook.container:89`); database: `/surreal is-ready` (`open-notebook-surrealdb.container:73`) |
| Env | `scripts/open-notebook.env` (repo, knobs) then `/home/dank/Containers/open-notebook/env` (host, secrets, mode 600) — two `EnvironmentFile=` lines per unit, repo file **first** because the last file wins |
| Deployer | `deploy_open_notebook` in `scripts/deploy-remote.sh`: seed secrets → ensure both digest pins are in local storage → install both units + the network → `mkdir -p /home/dank/media/podcasts` → restart database, wait for its container health → restart app → `wait_for_health open-notebook http://127.0.0.1:5056/health 180` |
## What the image actually runs
One container, three processes under supervisord (`supervisord.conf` upstream): `uvicorn api.main:app --port 5055` (`:12`), the job worker (`:25`), and the Next.js frontend (`:39`). Two consequences the config depends on:
- **One unit owns the whole app.** There is no separate worker or frontend container to scale or restart, and the frontend refuses to start until the API answers (`wait-for-api.sh && node server.js`, `supervisord.conf:39`) — so gating the API gates the UI.
- **The worker's concurrency comes from the environment**, not a flag baked into the image: `--max-tasks ${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5}` (`supervisord.conf:25`). Setting it in the env file is a real change, not decoration.
Also from the image itself: `WORKDIR /app` and `EXPOSE 8502 5055` (`Dockerfile:84`, `:126`), **no `USER` line** (the app runs as container root, which in this host's rootless map lands as `dank` — the [[Seerr]] UID note measured `0 → 1000` on this machine), and `curl` **and** `ffmpeg` installed in the runtime layer (`Dockerfile:74-76`) — curl makes the health probe possible without a shell wrapper, ffmpeg is how the per-line clips become one file.
`DATA_FOLDER` is hard-coded `"./data"` (`open_notebook/config.py:5`) and `PODCASTS_FOLDER` is `"./data/podcasts"` (`:19`); the database stores each episode's audio path **relative** to that folder (`episodes/<uuid>/audio/<uuid>.mp3`), and the download route is `GET /api/podcasts/episodes/{id}/audio`. That is why covering one subdirectory with a bind mount is sufficient — nothing in the database has to change if the mount moves.
## Provisioning the providers (after first boot; UI or REST, never a unit)
Credentials are stored in the database, encrypted, and are **not expressible in env or a Quadlet file**. The sequence is one-time, in **Manage → Models**, and both entries use provider type **OpenAI Compatible** (upstream: any server that implements `/v1/audio/speech` works this way — `docs/5-CONFIGURATION/local-tts.md:5`).
1. **TTS credential** — base URL **`http://frank-tts:7772/v1`**. The `/v1` is part of the URL: the client posts to `<base_url>/audio/speech`, so a base URL without `/v1` 404s on every dialogue line (`local-tts.md:72-80`, the base-URL table, always carries `/v1`). `frank-tts` resolves by container name because the app is attached to `dankclaw_default`, the network frank-tts's unit joins (`dankclaw/voiceservice/quadlet/frank-tts.container:51-52`). If frank-tts is ever moved off that network, the published-port form is `http://192.168.0.143:7772/v1` — a published port is dialable at the host's LAN address from any netns. The API-key field stays empty; frank-tts does not authenticate.
2. **LLM credential** — base URL **`http://host.containers.internal:7771/v1`**, model row `frank-zealot`. frank-zealot is a **host process** on `0.0.0.0:7771` (`ss -ltn` on Dankclaw shows the listener with no container owning it, 2026-10-18), and on this netavark host a bridge container reaches the host's LAN IP only at a *published* port — which is why Prometheus scrapes the same endpoint as `host.containers.internal:7771` (`shared/prometheus/prometheus.yml:89-111`, "not their own LAN addresses"). See [[Container Network Namespaces]]. `http://192.168.0.143:7771/v1` is the variant to try only if the gateway name ever stops resolving; it is expected to be refused from a bridge netns.
3. **Model rows are created by name.** Discovery is optional (see the next section): type the model id and confirm the add. One base URL per credential, so the TTS and LLM credentials are necessarily two separate configurations.
**Verify reachability before saving**, from inside the container, because both addresses are netns-dependent:
```bash
podman exec open-notebook curl -sS http://frank-tts:7772/health
podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/models
```
**The blocker to state plainly:** frank-tts is built, named and tested but **not deployed** — nothing answers on 7772 (`ss -ltn`, Dankclaw, 2026-10-18; [[Local TTS and Voice Cloning]] records the same census). open-notebook deployed today writes scripts and fails at the first voice line. frank-tts ships first, or podcast generation waits.
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
## Speakers and voices: 1–4, and the IDs are free text
A **speaker profile** carries one to four speakers (`docs/3-USER-GUIDE/creating-podcasts.md:27`), each with a personality and a **Voice ID that is typed exactly and passed through to the TTS request untouched** (`:29`) — there is no whitelist, no enum, no validation against a voice list. So the fleet's voice-pack names work verbatim:
| Speaker | Voice ID to type |
|---------|------------------|
| Speaker 1 | `narrator1` |
| Speaker 2 | `narrator2` |
| Speaker 3 | `narrator3` |
Those are the pack names frank-tts resolves from `/home/dank/voices` (`scripts/frank-tts.env` header: a voice is real exactly when `/voices/<name>/reference.wav` exists). Two traps, both from upstream's shipped defaults:
- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `narrator1`/`narrator2` pair (or a third/fourth) rather than editing the seeded ones in place.
- Two speakers with the same voice are hard to follow (`creating-podcasts.md:89`). Distinct packs per speaker is the point of the 1–4 shape.
## Two buttons that fail against frank-tts, and why that is cosmetic
- **Test Connection** probes `GET <base_url>/models` (`open_notebook/ai/connection_tester.py:208-229`). frank-tts serves exactly two GET routes — `/health` and `/v1/audio/voices` — and 404s every other path (`dankclaw/voiceservice/app.py:228-238`). The button reports a failure against a healthy server.
- **Discover Models** does the same `GET <base>/models` (`open_notebook/ai/model_discovery.py:26-28`) and finds nothing to list.
- The TTS test path also synthesises with a **hard-coded voice `alloy`** for `openai_compatible` (`connection_tester.py:328-336`), which is not a fleet voice pack and would be refused even if the probe reached frank-tts.
None of it blocks anything: model rows are added **by name** ("If a model isn't listed, type its id and click Add", `local-tts.md:82`) and the Voice ID field is free text. The failure is a missing `/models` route on a deliberately minimal shim, not a broken credential. The honest end-to-end test is a real generation job, or the `podman exec … curl /v1/audio/speech` call frank-tts's own seam documents.
## The CPU guard, and why both numbers are 1
Defaults are `TTS_BATCH_SIZE=5` and `OPEN_NOTEBOOK_WORKER_MAX_TASKS=5` (`docs/5-CONFIGURATION/environment-reference.md:129-130`) — five parallel synthesis requests from five parallel jobs. frank-tts spawns an engine process per request (about 8 GiB resident, 8 threads, from the frank-tts CPU trial of 2026-10-18; `scripts/frank-tts.env` pins `LLAMA_TTS_THREADS=8`), on a machine that also serves frank-dragoon and frank-zealot. Five of those is ~40 GiB and forty threads competing with two LLM endpoints for the same cores. `scripts/open-notebook.env:87-88` sets both to 1: one episode at a time, one voice line at a time. Upstream's own guidance is the same direction ("Set 1 for a local model on a single GPU", "Lower it … for rate-limited or single-threaded TTS servers").
The clock next to it: `ESPERANTO_TTS_TIMEOUT` (`:91`) is patience **per dialogue line**, not per episode — a three-minute episode is dozens of separate POSTs, and the tail of a long turn on a CPU engine goes past upstream's 300 s default. It is set to **660, one round-trip outside `SHIM_TIMEOUT=600`** in `scripts/frank-tts.env` (frank-tts's own per-request ceiling): nested budgets go inside-out — the same rule that file states for `LLAMA_TIMEOUT` — so a frank-tts give-up arrives labelled as frank-tts's error instead of the worker's own timeout, and re-measuring frank-tts cannot silently strand this number below it. An abandoned request does not cancel the engine process; it only strands the episode in "generating".
## Secrets, and the difference between blank and unset
`OPEN_NOTEBOOK_PASSWORD` unset means **authentication off** (`environment-reference.md:46`) — every API route, including the episode-audio download, answers anyone who can reach the published port. That is a real exposure on a LAN-published service with no vhost and no guard chain in front of it, which is why `deploy_open_notebook` refuses to start either unit unless the seeded host file carries a value for all four secret keys (`OPEN_NOTEBOOK_ENCRYPTION_KEY`, `OPEN_NOTEBOOK_PASSWORD`, `SURREAL_PASSWORD`, `SURREAL_PASS`). `OPEN_NOTEBOOK_ENCRYPTION_KEY` is separately load-bearing: it is **required to save any provider credential**, and changing it later makes stored keys undecryptable (`:44`).
The tracked file declares the secrets **empty** and the host file supplies them; podman applies `--env-file` arguments in order and the last one wins (verified against the generated `ExecStart` line, 2026-10-18). Reordering the two `EnvironmentFile=` lines in a unit would blank every credential — that is the trap the seam's ordering assertion exists for.
The database has the same trap in a different spelling: the app reads `SURREAL_PASSWORD`, the `surreal` CLI reads `SURREAL_PASS` (its `start --help` prints `[env: SURREAL_PASS=]`; measured against the pinned digest). One value, written twice, by the seed function — so the pair cannot be given different root passwords. Both arrive through the environment, **not** through `Exec=`: `--pass` on the command line would put the database password in `/proc/<pid>/cmdline`, readable by any `ps` on the host. Measured with the pinned image: env-only credentials produce `Initialising credentials user=…`, the root user is created on first boot, a correct login authenticates and a wrong password is refused — no `--user`/`--pass` flags needed, and no `--bind` either, because the image's own `SURREAL_BIND=0.0.0.0:8000` is what makes the RPC reachable from another container at all.
## Episode destination: interim, and reversible (issue #7)
**Episode files currently exist in two places by design.** The app's own state stays in the `open-notebook-data` named volume; finished audio additionally lands on the host, because `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts` (`open-notebook.container:88`). That is the **interim** answer to where episodes live: the destination is an open decision tracked as **Gitea issue #7, labelled `human-only` — agents must not work it.** Until Frank settles it, episode MP3s are ordinary files in his home directory, readable without `podman unshare`, which the database volume would require (measured with the pinned SurrealDB image).
Reversibility is one line: delete the `Volume=/home/dank/media/podcasts:/app/data/podcasts` line and the episodes go back to living solely inside the named volume. Nothing else in the config assumes the bind. See [[Storage Policy]] for the repo's storage conventions this interim state deliberately bends.
## Deploying, and the digest-pull step
```bash
# on Dankclaw, as dank — the pins are Pull=never, so fetch them first
podman pull docker.io/surrealdb/surrealdb:v2@sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723
podman pull docker.io/lfnovo/open_notebook:v1-latest@sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212
```
`deploy_open_notebook` runs the same two pulls when the image is absent from local storage, so the manual step is only needed to pre-seed an offline host or to bump a pin. Then `scripts/deploy.sh --target dankclaw --service open-notebook`. Both `Pull=never` and the pin comment live in the unit files; upgrading means **editing the digest**, not repulling a tag — see [[Deployment Flow]].
## Rollback
- **Nothing deployed (today's state):** revert the commits. There is no host state to undo — no unit, no container, no volume, no `/home/dank/media/podcasts` directory beyond what the deployer would create.
- **Deployed:** `systemctl --user stop open-notebook.service open-notebook-surrealdb.service`, then move the three files (`open-notebook.container`, `open-notebook-surrealdb.container`, `open-notebook.network`) out of `~/.config/containers/systemd/` into `~/.AGENTTRASH/` — never `rm` — and `systemctl --user daemon-reload`. The generated units disappear with their sources.
- **Kept on purpose:** the `open-notebook-data` and `open-notebook-db` named volumes and `/home/dank/media/podcasts`. Episode audio and the notebook database are user data; removing them is a separate, deliberate act, and episode files are host files precisely so they can outlive the container.
- **Homepage:** the card is `wip: "down"` + `showLink: false`, so retiring the service needs no card surgery beyond setting it back to that state.
## Citations
- Repo: `dankclaw/open-notebook/quadlet/{open-notebook.container,open-notebook-surrealdb.container,open-notebook.network}`, `scripts/open-notebook.env`, `dankclaw/.env.example`, `scripts/deploy-plan.py`, `scripts/deploy-remote.sh` (`seed_open_notebook_env`, `ensure_pinned_image`, `deploy_open_notebook`), `scripts/deploy.sh`, `scripts/test-changed.sh`, `tests/open_notebook.bats`, `shared/homepage/services.json`, `dankclaw/seerr/quadlet/seerr.container` (the 5055 collision), `dankclaw/voiceservice/{quadlet/frank-tts.container,app.py}`, `scripts/frank-tts.env`, `shared/prometheus/prometheus.yml`
- Upstream `lfnovo/open-notebook` @ main, read 2026-10-18: `docker-compose.yml:3,10,11,18,27,29-30,39-43`, `Dockerfile:69-76,84,126,140,158`, `supervisord.conf:12,23-25,39`, `api/main.py:73,227-230,239-241,414`, `api/credentials_service.py:418-420,552`, `api/routers/credentials.py:426-429,433-437`, `open_notebook/config.py:5,19`, `open_notebook/podcasts/audio_paths.py`, `open_notebook/ai/connection_tester.py:208-238,326-336`, `open_notebook/ai/model_discovery.py:26-28`, `docs/3-USER-GUIDE/creating-podcasts.md:13,27,29,33,89`, `docs/5-CONFIGURATION/environment-reference.md:44,46,106,110,129,130`, `docs/5-CONFIGURATION/local-tts.md:5,45,72-82,89`
- Registry (2026-10-18): `lfnovo/open_notebook` tags (`v1-latest`, `latest`, the `-single` set) and manifest-list digests; `surrealdb/surrealdb` `v2` → 2.7.0
- Local measurements (2026-10-18, workstation, pinned digests): `surreal start --help` (credential and bind options carry `[env: SURREAL_*]`); running the pinned SurrealDB digest with env-only credentials (credential init in the log, correct login authenticates, wrong password refused, `/surreal is-ready` → `OK` exit 0, named volume owned by the mapped uid); the v1-compatible Quadlet generator dry-run of all three unit files (two `--env-file` in order, two `--network` in order, named volume + host bind, both health probes, `--pull never`)
- Host evidence (Dankclaw, 2026-10-18, read-only): `ss -ltn` (`127.0.0.1:5055` held by seerr, `0.0.0.0:7771` a host listener, nothing on 7772/8502/5056/8000), `podman ps -a` (no open-notebook container, `dankclaw_default` present), subnet registry from `dankclaw/open-webui/quadlet/open-webui.network`
@@ -0,0 +1,251 @@
---
type: reference
description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes."
tags:
- dankclaw
- frank-tts
- voices
- tts
timestamp: 2026-10-18
---
# Voice Packs
A **voice pack** is the unit of identity for **frank-tts**: one directory under
`/home/dank/voices`, named by the voice, carrying the reference audio that makes
the name real plus the pinned engine's own speaker artifacts and a provenance
record. `voice=narrator1` on `POST /v1/audio/speech` resolves to
`/home/dank/voices/narrator1/reference.wav` and to nothing else — a name without
that file is a 400 that lists the names that do exist, because a silent fallback
voice is how a wrong-voice endpoint stays wrong for weeks.
**Host:** Dankclaw (192.168.0.143) — **Pack root:** `/home/dank/voices`, host
state that stays out of the repo, mounted read-only at `/voices` by the façade
unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in
`scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:**
arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]].
Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade
container runs as uid 10000 and reads the mount from outside its own uid map.
## The required files
| File | What it is | Who reads it |
|------|------------|--------------|
| `reference.wav` | the crop: 24 kHz mono `pcm_s16le`, **18–25 s** of clean single-speaker audio | the façade (it is the existence gate for the name) and the engine's boot warmer, which base64s it and registers the voice with no transcript |
| `reference.spk` | the engine's x-vector: **8,192 B** = `enc_dim` 2048 × f32 | the engine, for the ICL/`--ref-spk` path |
| `reference.rvq` | the engine's RVQ code stream: **5,456–6,864 B**, 16 codebooks × T frames packed at 11 bits = 22 B per 12.5 Hz frame | the engine, for the ICL/`--ref-rvq` path |
| `manifest.json` | provenance: source path and sha256, crop start/end offsets, sample rate, duration, the tool and engine commit that made the artifacts, and the `voice` name | humans, and the builder itself on its second gate assertion |
`qwen-codec.log` rides along with the extractor's own record of the run; the
façade ignores anything it does not name and never deletes anything.
Division of enforcement, which is deliberate and not a gap: the **façade** gates
on `reference.wav` because that is the only engine-agnostic term (see the
voice-pack law in `scripts/frank-tts.env`); the **builder** is what refuses to
emit a pack without its engine artifacts; the **engine** registers a voice from
the WAV alone because its own `POST /v1/audio/voices` refuses a lone embedding —
it wants `wav_b64` or **both** `spk_b64` and `rvq_b64`. So the artifacts are in
the pack to make ICL a per-voice opt-in, not to boot it.
## How a pack is built
```bash
# everything except the extractor: gate, crop, probe, plan, the manifest it would write
scripts/build-voice-pack.sh --dry-run narrator1 /home/dank/voices
# the real thing, on the host that holds the engine build
scripts/build-voice-pack.sh narrator1 /home/dank/voices
```
Crops come from `scripts/voice-packs.env` — per voice: source path, offset,
duration, and the source's pinned sha256 — and any of them can be overridden on
the command line (`--source`, `--offset`, `--duration`, `--sha256`, `--tool`).
`--all` walks `VOICE_PACKS`. The builder assumes no GPU, no container, no fleet
endpoint and no network: `ffmpeg` cuts the crop, the pinned extractor makes the
artifacts, and both run `nice -n 19` so a build never competes with the text
endpoints on that box.
The extractor is `qwen-codec` in the `qwentts.cpp` tree at
`VOICE_PACK_ENGINE_COMMIT` (`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`). Its
usage text, read out of that tree and not guessed:
```
Usage: qwen-codec --model <gguf> [-i <input>] [--talker <gguf>] [--format <fmt>]
Required:
--model <gguf> Codec GGUF (qwen-tokenizer-12hz-*.gguf)
Optional:
--device <name> Compute device: CUDA0, Vulkan0, MTL0, CPU...
(default: best available)
-i <path> Input. WAV -> encode, .rvq -> decode
--talker <gguf> Talker GGUF (Base only). Encode also extracts the speaker
embedding and writes it next to the .rvq as a .spk file
--format <fmt> WAV output format: wav16, wav24, wav32 (default: wav16)
Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav.
```
Three consequences shape the builder. **There is no `-o`**, so the pack's WAV is
staged under its final stem and the artifacts land beside it with nothing to
rename. **`--talker` must be a _Base_ talker** — the customvoice and voicedesign
GGUFs carry no speaker encoder and the tool says `has no speaker encoder (Base
only)`. **Encode truncates to the hop boundary**, so the `.rvq` size is arithmetic,
not chance: `22 × floor(samples / 1920)` bytes, which is where the recorded
5,456–6,864 B window comes from and why the **effective crop window is
19.84–25 s** even though the WAV law says 18–25 s. A crop shorter than that is
refused before the extractor runs, with the number in the message.
The build is staged and published, never assembled in place: `reference.wav` is
the file the façade reads, so it moves into position **last**, after the
artifacts and the manifest. A failed extractor therefore leaves a pack directory
that is not a voice. Nothing is ever deleted — an existing pack is refused unless
`--force`, and `--force` moves the whole old directory to
`/home/dank/voices/.AGENTTRASH/<voice>-<UTC>/` and builds fresh beside it.
Measured on the shipped `narrator1` crop (2026-10-18, `--dry-run` against the
real source): 98.739–118.636 s → `955,378 B` of 24 kHz mono `pcm_s16le`, 19.897
s, planned `.rvq` 5,456 B — the same byte counts the trial produced by hand, and
the extractor itself costs 1.8–2.1 s and 1.3–1.5 GiB peak per voice.
## The 10-second gate
Project-wide source-material invariant: **the first 10 seconds of any sample
file used to generate voices is forbidden material.** An offset that starts
inside it is refused with exit 1 and a message that names the invariant, before
a directory is created:
```
ERROR: source-material invariant violated: the crop must start past the first
10 s of the source file, and 'narrator1' asks for 4.5 s into … — nothing was
written.
```
The same gate is then asserted **from the manifest** that was just written —
`.crop.offset_seconds >= .invariant.forbidden_source_seconds`, plus the recorded
sha256 against the source it claims — so a pack cannot reach disk carrying a
forbidden crop even if the offsets in the env file are edited into nonsense.
`10` is not a knob: `scripts/voice-packs.env` carries the offsets, the sample
rate, the windows and the engine facts, but the forbidden region is a constant in
the builder. `tests/voice_pack_unit.bats` pins both halves (a refusal that
creates nothing, and one legal crop that builds the whole pack) and applies the
invariant to every crop in the tracked pack list.
## Adding a fourth voice
1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop
offset is a claim about one exact file, so the digest goes in the env with it.
2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once:
`ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`,
build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest
on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the
`.rvq` falls under the floor.
3. Add the four lines to `scripts/voice-packs.env` and the name to
`VOICE_PACKS`:
```
VOICE_PACKS="narrator1 narrator2 narrator3 narrator4"
VOICE_PACK_NARRATOR4_SOURCE=/home/dank/voice-sources/audiosample4.m4b
VOICE_PACK_NARRATOR4_OFFSET=142.31
VOICE_PACK_NARRATOR4_DURATION=24
VOICE_PACK_NARRATOR4_SHA256=<64 hex>
```
The name must be lowercase `[a-z0-9._-]`: the façade matches the directory
name verbatim while `tts-server` lowercases names internally, so an uppercase
name would be reported one way by the façade and another by the engine. The
builder refuses one anyway.
4. `scripts/build-voice-pack.sh --dry-run narrator4 /home/dank/voices`, read the
plan, then drop `--dry-run`.
5. `./tests/run.sh` (the pack-list invariant test covers the new row for free).
6. Register it with the running engine. The engine's voice registry is
memory-only and warm-on-boot, but a restart re-decodes **every** voice (6–8 s
each), so adding one pack uses the hook instead:
```bash
podman exec frank-tts-engine /app/warm-voices.sh
curl -s localhost:7772/v1/audio/voices # the façade's list, not the engine's
```
The hook walks `/voices`, registers any pack with a `reference.wav` it has not
seen, and logs `voice <name> cold=…ms warm=…ms` to the unit's journal. The
rootless unit restart — `systemctl --user restart frank-tts-engine.service` —
is for when the unit or its env changed, not for a new pack.
## When the extractor's output size changes on an engine bump
The builder treats the two artifact sizes as properties of the pinned engine, so
an engine bump that changes them **fails the build**: `reference.spk is N B,
expected 8192 B` or `reference.rvq is N B, expected …`. That is the failure being
loud, not a bug. Work through it in order:
1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec`
prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , …
codebooks @ … Hz` on every run, and the pinned tree's
`tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`;
`codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`.
2. Decide whether the new number is intended. A changed `enc_dim` means the old
`.spk` files are unreadable by the new engine, so **every** pack must be
rebuilt, not just the failing one.
3. Update the four arithmetic lines in `scripts/voice-packs.env`
(`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`,
and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and**
`VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`,
`VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes
without moving the commit is the mistake to avoid.
4. Rebuild all three packs with `--force` (the retired directories keep the old
bytes readable beside their replacements until a human clears them), then
restart the engine rather than using the warm hook: the hook only registers
packs it has **not** seen, and every voice here is already registered against
the old artifacts. `systemctl --user restart frank-tts-engine.service`
re-warms them all from the new bytes.
5. `./tests/voice_pack_unit.bats` — the stub in that seam reproduces the packing
law, so it needs the same arithmetic update as the env file. The seam's
engine-bump test is exactly this scenario with a wrong `.spk` size, and its
assertion — no `reference.wav`, no manifest — is the property that matters.
## Host state (census 2026-10-18, read-only)
- `/home/dank/voices/` holds `narrator1`, `narrator2`, `narrator3`, each with
`reference.wav`, a `reference16k.wav`, and a `manifest.json` **in the trial's
own schema** (`kind: voice-reference-pack`, a silence-scan record). They carry
**no** `reference.spk` / `reference.rvq`: the trial's artifacts live in its own
work directory (`/home/dank/voice-audition/qwentts/voices/`, and only for
narrator1 and narrator2). The builder's first real run is therefore a
migration, and it should be a `--force` run per voice so the old directories
are retired rather than left half-updated.
- The three source files (`/home/blank/audiosample.mp3`,
`audiosample2.m4b`, `audiosample3.m4b`, digests pinned in
`scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the
builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so
they have to be copied there once before a build.
- `VOICE_PACK_TOOL` points at the trial's build tree
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
at exactly `5916bf7c`), and the two GGUFs at
`/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's
download script puts them — that directory does not exist yet, so until it
does the models have to come from the trial's `models/` directory.
- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs
is possible on that host while serving them is not.
## Known gaps, on purpose
- **No transcript in the manifest.** ICL wants a `ref_text` that matches the
crop, and the trial's machine transcripts came back wrong ("Demon Souls" for
*Dark Souls*). Transcribing on the host is not a fix, so the pack carries none
and the engine registers the WAV-only voice, which is what its boot warmer
does anyway. When ICL becomes a per-voice opt-in, the curated transcript
belongs in this manifest — a human-supplied field, not something the builder
can derive.
- **No 16 kHz variant.** The packs the trial left behind carry
`reference16k.wav` for the `llama-tts-cli` rollback engine; arm A is native
24 kHz, so the builder emits one reference. `--force` retires the whole
directory including that variant, so if the rollback engine is ever revived:
`ffmpeg -i reference.wav -ar 16000 -c:a pcm_s16le reference16k.wav`.
- **No loudness pass.** The trial's pipeline also measured and normalised
(two-pass `loudnorm`, `astats`, `ebur128`, spectrograms) to a `-16 LUFS` spec.
None of that is in the pack law, which asks for 24 kHz mono inside the
duration window, so the builder does only that; a future engine that is
loudness-sensitive should get the pass as an explicit, documented step.
@@ -1,7 +1,7 @@
--- ---
type: research type: research
title: Local TTS and Voice Cloning title: Local TTS and Voice Cloning
description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the full local AI audio stack, and a recommended homelab deployment. description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts and the arm A decision, the full local AI audio stack, and a recommended homelab deployment.
tags: tags:
- research - research
- tts - tts
@@ -10,7 +10,7 @@ tags:
- stt - stt
- vllm-omni - vllm-omni
- kokoro - kokoro
timestamp: 2026-10-18T00:00:00Z timestamp: 2026-10-18T16:05:00Z
--- ---
# Local TTS and Voice Cloning # Local TTS and Voice Cloning
@@ -47,13 +47,16 @@ greenfield: every fleet endpoint declares `input: [text, image]`
local server needs a speech-role entry, not a patch. local server needs a speech-role entry, not a patch.
- **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue - **Fleet plan (2026-10-18): CPU-only on Dankclaw, not the 3090.** Gitea issue
#2 rules out every GPU option (the 3090 has carried frank-zealot since #2 rules out every GPU option (the 3090 has carried frank-zealot since
2026-10-17; both R9700s are frank-archon's), so the shape is a VoxCPM2 GGUF 2026-10-17; both R9700s are frank-archon's). The shape is a TTS engine on
engine on **8880** behind a voice-registry shim on **8890** **8880** behind **frank-tts**, the fleet's speech endpoint on **7772**
(`scripts/voiceshim.env`, `dankclaw/voiceservice/`) that speaks OpenAI (`scripts/frank-tts.env`, `dankclaw/voiceservice/`): one OpenAI
`/v1/audio/speech` and resolves voices by name. Nothing is deployed yet — on `/v1/audio/speech` façade that resolves a voice *by name* and never shows a
2026-10-18 `voiceshim` has no unit installed and nothing answers on 8880 or path. Nothing is deployed — no `frank-tts` unit and no container exist on the
8890. STT (Parakeet TDT 1.1B, CC-BY-4.0) stays on CPU. **No RTF or quality host and nothing listens on 7772 or 8880 (checked 2026-10-18). STT (Parakeet
number has been measured for this shape**, so this page states none. TDT 1.1B, CC-BY-4.0) stays on CPU. **The engine is chosen: arm A**
`qwentts.cpp` **resident `tts-server`, 1.7 B talker** — measured in
*Measured CPU engine A/B* below and picked on the live-output criterion in
[Decision: arm A for live output](#decision-arm-a-for-live-output-2026-10-18).
## Can Kokoro learn a new voice? ## Can Kokoro learn a new voice?
@@ -133,11 +136,11 @@ accepts only its built-in voice list, so a custom pack needs a patch.
Paths for a Frank-cloned voice (2026-10-18): Paths for a Frank-cloned voice (2026-10-18):
1. **Local server + speech role, no patch.** Register the local engine (or the 1. **Local server + speech role, no patch.** Register frank-tts (port 7772) as
8890 voice shim) as a provider model on the `openai-speech` API and point a provider model on the `openai-speech` API and point `modelRoles.speech` at
`modelRoles.speech` at it; the shim turns a voice *name* into a reference it; the façade turns a voice *name* into that pack's reference clip and
clip and forwards it as base64 `reference_audio` forwards it as base64 `reference_audio` (`scripts/frank-tts.env`,
(`scripts/voiceshim.env:19-35`). Not deployed as of 2026-10-18. `TTS_BACKEND=omni-http`). Not deployed as of 2026-10-18.
2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean 2. **xAI console custom voice — works today, zero code.** Record 2–3 min clean
speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech speech (≤120 s, 24 kHz 16-bit mono WAV) → console → `voice_id` → speech
tool. Free (≤30 voices), cloud, US-only. tool. Free (≤30 voices), cloud, US-only.
@@ -206,29 +209,161 @@ The CPU shape does not need a Python server at all:
`VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own `VoxCPM2-Acoustic-F16.gguf`, Apache-2.0 card) or from the fork's own
`tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`. `tools/omni/voxcpm2/convert_voxcpm2_to_gguf.py`.
Nothing has been measured on this fleet for any of these engines — no RTF, no Arm B (`llama-tts`) and arm A (`qwentts.cpp`) have both been measured on this
latency, no MOS. That absence is the finding, not an omission. box — RTF, first-audio latency and RSS are in the next section. The other
servers in the table above (vLLM-Omni, Kokoro-FastAPI, CosyVoice, F5-TTS,
GPT-SoVITS) remain unmeasured here; that absence is still the finding.
## Measured CPU engine A/B (2026-10-18)
Both candidate upstreams for frank-tts were measured on Dankclaw's CPU — 8
threads, no GPU touched, three runs per arm, seed 42, identical text, against
the fleet's own audition voice packs.
- **Arm A — [`ServeurpersoCom/qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp)**
at commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` (2026-10-16), weights
from [`Serveurperso/Qwen3-TTS-GGUF`](https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF):
`qwen-talker-1.7b-base-Q4_K_M.gguf` (1,219,245,248 B),
`qwen-talker-0.6b-base-Q4_K_M.gguf` (628,905,056 B),
`qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B) — byte counts verified
against the Hub API. Builds with zero compiler warnings; `GGML_BLAS=OFF` is
forced on this host (only `libopenblas.so.0` exists, no dev headers, so the
vendor `buildcpu.sh` cannot configure).
- **Arm B — mainline `llama-tts`** with `--tts-speaker-file`, i.e. the
`llama-tts-cli` backend `scripts/frank-tts.env` already implements.
RTF is **seconds of compute per second of audio** (lower is better; below 1 is
faster than real time).
| Arm | RTF | Run spread | Peak RSS |
|---|---|---|---|
| A · 0.6 B talker, CLI | 0.44 | ±0.002 | 3.4 GiB |
| A · 1.7 B, CLI, uncloned | 0.57–0.58 | ±0.006 | 4.7 GiB |
| A · 1.7 B, CLI, embedding-only clone | 0.63 | ±0.02 | n/m |
| A · 1.7 B, CLI, ICL clone | 0.92 | ±0.01 | n/m |
| A · 1.7 B, resident `tts-server`, warm | 0.55–0.57 | ±0.01 | 5.7 GiB |
| A · same, ICL voice warm / **cold (first request)** | 0.67 / **1.65** | one-shot | 5.7 GiB |
| B · mainline `llama-tts` | 0.93–0.99 | ±0.25 | n/m |
**Read RTF only within an arm.** Conditioning changes clip length — the same
seed yields 18.00 s of audio embedding-only and 15.12 s under ICL — so the
denominator is the arm's own audio duration.
Arm A's resident server is ~1.7× faster than arm B and about 40× tighter in
run-to-run spread, and it already serves the HTTP shape frank-tts's façade
speaks: `/health`, `/v1/models`, `/v1/audio/voices` (GET/POST/DELETE),
`/v1/audio/speech` (`wav`/`pcm`), chunked streaming with the first audio byte
at 86–451 ms and a server-side time-to-first-audio of 94–112 ms. Arm B has no
HTTP path at all — that is what the façade's CLI backend exists for.
Four laws came out of the trial, each worth a line of configuration:
- **Boot-warm every voice.** The first request for a registered voice decodes
the whole reference to snapshot codec state: 6.2–8.0 s, first audio byte at
5,530 ms. Later requests restore that state in 0.5 ms. An engine unit must
run one throwaway synthesis per voice at start, or the first request of the
day pays five seconds.
- **Speaker artifacts are reusable and exact.** `qwen-codec` turns a reference
WAV into `.spk` (8,192 B x-vector) + `.rvq` (5,456–6,864 B, 16 codebooks × T,
11-bit packed) in 1.8–2.1 s at 1.3–1.5 GiB peak; replaying them yields a
**byte-identical** WAV and saves 1.4 s per request. That pair is the
engine-artifact term the voice-pack law in `scripts/frank-tts.env` reserves a
slot for — arm A pins it as `.spk` + `.rvq` beside `reference.wav`.
- **Streaming is not bit-identical to buffered**, whatever the upstream README
claims: same text, seed and voice, 92.2 % of samples differ (peak delta
1139/32768, −41.6 dB RMS — inaudible). A fixed seed does not pin output
across `response_format`, so never mix the two paths in a cache or a golden
fixture.
- **A voice cannot be registered from an embedding alone**: `POST
/v1/audio/voices` accepts `wav_b64` or **both** `spk_b64` and `rvq_b64`
(`src/tts-server.h:480`), so an embedding-only voice has to be sent as the
WAV, or the lone-`spk` path has to be upstreamed.
**Likeness is not measurable at this resolution.** x-vector cosine against each
speaker's own reference: ICL 0.9886 / 0.9901, embedding-only 0.9844 / 0.9812,
uncloned 0.9228 — but **two different speakers score 0.9497** on the same
metric. The whole clone effect lives in a 0.04-wide band and the
ICL-over-embedding gap is inside noise, so the four audition clips
(`/home/blank/voice-audition/qwentts-*.mp3`, each against its `armB-*.mp3`
mainline partner, guide in `LISTEN-engine.md`) are the instrument. **The
operator heard them on 2026-10-18 and passed every clone**, which removed the
last axis measurement could not reach; the engine was then picked on
live-output grounds ([Decision](#decision-arm-a-for-live-output-2026-10-18)).
The cosine
extractor is still worth keeping: it is engine-neutral (it scores mainline
output through the same encoder) and it separates an uncloned voice cleanly.
Two operational details. The engine takes 8 threads and exposes **no `-t`
flag** — `src/backend.h` uses `hardware_concurrency()/2` and prints the choice
at load, so pinning a cpuset on the quadlet changes that number implicitly. And
ICL needs a reference transcript that did not exist: the fleet has no ASR
endpoint and no audio input, so the crops were transcribed with
`vosk-small-en-us` inside the work dir, and two of three transcripts came back
suspect ("demon souls" for *Dark Souls*) — a reason to curate `ref_text` into
the pack manifest instead of transcribing on the host.
## Decision: arm A for live output (2026-10-18)
Every audition pair passed, so likeness stopped being the tiebreak. The
question became **which engine is better for live output** — the harness
speaking a reply, an alert read aloud, a click-to-speak — and on that criterion
RTF only has to stay under 1.0 with margin: what separates the arms is **time to
first audio** and the **spread**, because an arm whose RTF can cross 1.0 stalls
a live stream mid-sentence.
| | arm A · `qwentts.cpp` `tts-server` | arm B · mainline `llama-tts` |
|---|---|---|
| first audio byte | 86–451 ms | model load, per request |
| warm RTF | 0.55–0.67 | 0.93–0.99 |
| run spread | ±0.01 | ±0.25 |
| HTTP | serves `/v1/audio/speech` | none — façade CLI only |
| RAM | 5.7 GiB resident | per-request, freed after |
**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine`
container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.**
Three consequences are configuration, not opinion: the unit **boot-warms every
voice** (5.5 s cold first byte otherwise), it **does not pin a cpuset** casually
(threads are `hardware_concurrency()/2`, no flag), and every pack registers with
the **WAV** because the engine rejects a lone embedding.
Two follow-ups stay open on purpose, and neither blocks the deploy:
- **Talker size.** The 0.6 B arm measured 0.44 RTF at 3.4 GiB and has never
been auditioned — every clip above is 1.7 B. It is the lever for a CPU
co-tenancy problem, not a preemptive saving.
- **ICL vs embedding-only.** Both passed the ear test. Embedding-only is the
shipped default: 0.57–0.63 vs 0.67 warm, and it needs no `ref_text`, which
this fleet cannot produce honestly (the ASR crops came back suspect). Packs
still carry `.spk` + `.rvq`, so ICL remains a per-voice opt-in for batch work
like the podcast render, where total time matters more than first byte.
## Recommended deployment (Dankclaw) ## Recommended deployment (Dankclaw)
As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090 As decided in Gitea issue #2, this stays **CPU-only on Dankclaw**: the RTX 3090
has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's, has carried frank-zealot since 2026-10-17 and both R9700s are frank-archon's,
so no card may be asked to co-host TTS. The shape is one engine plus a thin so no card may be asked to co-host TTS. The shape is one engine plus one
voice shim, and **nothing here is deployed as of 2026-10-18** — `systemctl façade, and **nothing here is deployed as of 2026-10-18**: `frank-tts` has
--user is-enabled voiceshim` reports `not-found` and nothing answers on 8880 or neither a unit nor a container on the host, and nothing listens on 7772 or
8890. 8880.
| Service | Image / install | Port | CPU | Notes | | Service | Image / install | Port | CPU | Notes |
|---|---|---|---|---| |---|---|---|---|---|
| TTS engine | `llama-tts-server` (`tc-mb/llama.cpp-omni`) on a VoxCPM2 GGUF; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 | engine decision still open | OpenAI `/v1/audio/speech`; `n_gpu_layers` is there for a GPU boot | | TTS engine | **arm A `qwentts.cpp` `tts-server`** on the Qwen3-TTS GGUFs (decided 2026-10-18, above); arm B mainline `llama-tts` stays in the repo as the `llama-tts-cli` rollback; `vllm/vllm-omni` + Qwen3-TTS only if a GPU is ever deliberately freed | 8880 (container-internal) | open, evidence above | arm A serves OpenAI `/v1/audio/speech` itself and boot-warms each voice; arm B has no HTTP path and runs through the façade's CLI backend |
| Voice shim | `dankclaw/voiceservice/` (stdlib container) + `scripts/voiceshim.env` + `dankclaw/voiceservice/quadlet/voiceshim.container` | 8890 | yes | voice *name* → `reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) | | frank-tts | `dankclaw/voiceservice/` (stdlib container) + `scripts/frank-tts.env` + its quadlet | 7772 | yes | voice *name* → `/voices/<name>/reference.wav`, `voice`/`voice_id` on the wire, mp3 re-encode; backends `omni-http` (forwards base64 `reference_audio`) and `llama-tts-cli` (`--tts-speaker-file`, `nice 19`) |
| STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) | | STT (English) | Parakeet TDT 1.1B behind a FastAPI wrapper, or whisper.cpp server | 8882 | yes | CC-BY-4.0; LibriSpeech-clean WER 1.39 (model card) |
| STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT | | STT (multilingual, opt.) | faster-whisper `large-v3` int8 | 8883 | yes | MIT |
| VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B | | VAD | Silero (embedded) | — | yes | MIT; model file 2 327 524 B |
8880/8890 are what `scripts/voiceshim.env:35,49` carry (`TTS_UPSTREAM= 7772 is frank-tts's host port — the next slot in the fleet's model-endpoint
http://host.containers.internal:8880`, `SHIM_PORT=8890`); 8882-8883 for STT family (7770 frank-dragoon, 7771 frank-zealot). The engine behind it keeps 8880,
keep clear of the fleet's LLM ports 7770/7771. but **not as a host-loopback publish**: netavark DNATs published ports only, and
a bridge container cannot reach a `127.0.0.1` bind on the host — that is the
lesson that 502'd hermes (`AGENTS.md`). `host.containers.internal`
(169.254.1.2) is the gateway, so it reaches a host service only when that
service binds a routable address; `scripts/frank-tts.env` names it as the
engine's address, which holds for a `0.0.0.0` bind and not for a loopback one.
The deployed pair therefore shares one quadlet network and the façade addresses
the engine by container name.
## Voice-creation workflow (the "from sample files" answer) ## Voice-creation workflow (the "from sample files" answer)
@@ -284,6 +419,10 @@ keep clear of the fleet's LLM ports 7770/7771.
https://huggingface.co/openbmb/VoxCPM2 https://huggingface.co/openbmb/VoxCPM2
- llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni · - llama.cpp-omni TTS server: https://github.com/tc-mb/llama.cpp-omni ·
https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md https://raw.githubusercontent.com/tc-mb/llama.cpp-omni/master/tools/omni/voxcpm2/README.md
- qwentts.cpp (arm A): https://github.com/ServeurpersoCom/qwentts.cpp at commit
`5916bf7c79dbb4e66fc059cdc507a79a23a28bb4` · weights
https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF (byte counts against the
Hub API, 2026-10-18)
- mainline `llama-tts`: - mainline `llama-tts`:
https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md · https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/tts/README.md ·
Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0) Pocket-TTS weights: https://huggingface.co/kyutai/pocket-tts (CC-BY-4.0)
+2
View File
@@ -79,6 +79,8 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
- [[Postgres]] — Gitea database backend - [[Postgres]] — Gitea database backend
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack) - [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress) - [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
## Infrastructure ## Infrastructure
+183
View File
@@ -2311,3 +2311,186 @@ every restart since the cutover.
Kokoro-enum-only — so "the agent talks in my voice" needs that provider row or Kokoro-enum-only — so "the agent talks in my voice" needs that provider row or
a `pi.registerProvider` extension, decided in phase 2. a `pi.registerProvider` extension, decided in phase 2.
## 2026-10-18 — TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion
- Frank listened to the four audition pairs (`qwentts-*.mp3` against their
`armB-*.mp3` mainline partners) and **passed every clone**. That deleted the
only axis the A/B could not measure, and the question was reframed: **which
engine is better for live output**. On that criterion RTF only has to stay
under 1.0 with margin — the discriminators are time-to-first-audio and
run-to-run spread, because an arm whose RTF can cross 1.0 stalls a live
stream mid-sentence.
- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`, 1.7 B
talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on 8880.**
The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env` stays in the
repo as the rollback. The numbers that carried it: first audio byte 86–451 ms
against a per-request model load, warm RTF 0.55–0.67 against 0.93–0.99, spread
±0.01 against ±0.25, and a native `/v1/audio/speech` against no HTTP path at
all. Three consequences are configuration, not preference: boot-warm every
voice, do not pin a cpuset (threads are `hardware_concurrency()/2`), register
every pack with its WAV (a lone embedding is rejected).
- Two follow-ups deliberately **not** decided: the **0.6 B talker** (0.44 RTF,
3.4 GiB, never auditioned — every clip is 1.7 B) and **ICL vs
embedding-only** conditioning. Embedding-only is the shipped default because
it is cheaper and needs no `ref_text`, which this fleet cannot transcribe
honestly; `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice
opt-in for batch work where total time beats first byte.
- **Addressing law corrected** in
[[Local TTS and Voice Cloning]]: the pair may not meet at a host-loopback
publish. netavark DNATs published ports only and a bridge container cannot
reach a host `127.0.0.1` bind — the lesson that 502'd hermes — so
`host.containers.internal:8880`, which `scripts/frank-tts.env` documents, is
correct only for a routable bind. The deployed pair shares one quadlet network
and the façade addresses the engine by container name.
- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`, 8 commits, 403 tests) is
**merged** as `1eb8b21` (Frank's OK, given in session): open-notebook +
SurrealDB quadlets on the private bridge `10.89.4.0/24`, the API published on
**5056** because seerr holds `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT`
moved 600 → **660** to sit one round-trip outside frank-tts's own
`SHIM_TIMEOUT`. Merged is not runnable: nothing listens on 7772 until the
voice stack lands, and the episode destination (`/home/dank/media/podcasts`)
stays interim pending Gitea #7.
- Still in flight, not in this branch: `feat/frank-tts-engine` — the arm A image
build, the quadlet pair on one network, the boot-warm step, deployer /
planner / `test-changed.sh` routing, homepage and docs. This branch is also
where the two-branch overlap got resolved: merging `main` here conflicted on
`log.md` alone (both branches appended an entry) and took `index.md`,
`deploy-remote.sh`, `deploy-plan.py` and `test-changed.sh` clean.
## 2026-10-18 — open-notebook: the podcast generator's config lands (nothing deployed)
- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — notes
and sources in, a multi-speaker episode out, script written by
**frank-zealot**, voices rendered by **frank-tts**, no cloud provider
anywhere in the path. Repo-side only: no host touched, no container pulled or
started. Runbook: [[Open Notebook]].
- **Two containers, not one.** `lfnovo/open_notebook:v1-latest` plus
`surrealdb/surrealdb:v2`, both **digest-pinned** with `Pull=never` and a
documented one-time `podman pull` — frank-dragoon's pinning style, because
`:latest` moving underneath a rootless service is how you get an unbootable
pod after an unrelated `systemctl-daemon-reload`. The registry round-trip is
what proved `v2` is SurrealDB **2.7.0** and that the app image is Debian
trixie with **curl and ffmpeg already inside** — which is how both health
probes can be argument-only: `curl -fsS …/health` (route is upstream's,
`api/main.py:239`, and it sits on the auth exclude list) and
`/surreal is-ready`, which I ran against the digest here: green in 2 s, and
it refuses to pass without `SURREAL_PASS` in the environment — a credential
that must never ride the command line, so `Exec=` carries flags only.
- **Host 5055 was already spoken for.** seerr has held `127.0.0.1:5055` since
2026-10-14 (live `ss -ltn` agrees today), so the API publishes
`0.0.0.0:5056:5055`. open-notebook's frontend derives its API URL from the
window origin when `API_URL` is unset — the upstream env reference spells the
reverse-proxy symptom — so a silent remap would hang the UI on a port nobody
listens on, and `API_URL=http://192.168.0.143:5056` is set accordingly. 8502
publishes on `0.0.0.0` because this service carries its own password (the
episode-audio route is unauthenticated), unlike the loopback-and-proxied
observability stack; **8000 is published nowhere**, where upstream exposes it
on loopback with root credentials, and the seam forbids it.
- **The CPU guard, pre-armed.** `frank-tts` spawns one ~8 GiB, 8-thread engine
per request on a box that also serves two LLMs, while open-notebook defaults
to `TTS_BATCH_SIZE=5` × `OPEN_NOTEBOOK_WORKER_MAX_TASKS=5` — a five-way
self-DoS. Both are pinned to `1` in `scripts/open-notebook.env`, and the
worker knob is only real because the image's own supervisord expands
`${OPEN_NOTEBOOK_WORKER_MAX_TASKS:-5}` (`supervisord.conf:25`). The TTS clock
goes to **660 — one round-trip outside** frank-tts's own `SHIM_TIMEOUT=600`, so
a shim give-up reads as the shim's error rather than the worker's; the seam
asserts the direction (`>`) against the frank-tts env instead of restating a
number. `ESPERANTO_LLM_TIMEOUT` stays at the app's 180 s default: frank-zealot
answers an outline or a transcript well inside it, and upstream says to keep it
under 600 so a slow call fails with a name on it.
- **What a credential needs in order to exist:** `OPEN_NOTEBOOK_ENCRYPTION_KEY`
must be set before the UI can save *any* provider credential, and
`OPEN_NOTEBOOK_PASSWORD` must stay **absent** to remain auth-off — the app
reads `os.environ.get(…) is not None`, so `PASSWORD=` boots a login form
asking for the empty string. The repo env file carries both keys empty; the
deployer seeds `/home/dank/Containers/open-notebook/env` (600) from the host
vault and **refuses to start the pair while either secret is blank**, because
a half-configured open-notebook stores no credentials and fails quietly.
- **Episode destination is interim, by decision.** The destination is **Gitea
issue #7, `human-only`** — agents must not work it. Until Frank settles it,
the app keeps its own named volume and `/home/dank/media/podcasts` is
bind-mounted over `/app/data/podcasts` (`DATA_FOLDER` is hard-coded `./data`),
so finished MP3s are ordinary files in his home directory instead of volume
contents needing `podman unshare` — the database volume demonstrably lands on
a mapped subuid, measured with the pinned image. One mount line; one deletion
undoes it.
- **Two UI buttons will fail, and that is cosmetic.** Test Connection does
`GET <base>/models` and the TTS probe synthesises voice `alloy`; frank-tts
serves `/health` and `/v1/audio/speech` only. Model rows are created *by
name* and the voice field is free text, so `narrator1/2/3` is typed in and
never discovered. Documented, not worked around.
- **Blocked on frank-tts, stated plainly:** phase 1 landed without deployer or
homepage registration and nothing listens on 7772 today, so a generation run
writes its script and then fails on the first voice. That is upstream's
creation-podcasts warning in fleet clothing.
- Suite green: **403/403** (`./tests/run.sh`, wall 67 s) — 19 tests in the new
`tests/open_notebook.bats`, structure only: pins, boot persistence, the
unpublishable database port, the 5055→5056 remap plus the `API_URL` that has
to match it, the env-file read order, the guard knobs, the health gate naming
`/health`, planner routing, the homepage card hidden while undeployed.
- Three deployer seams cost time and are now law-shaped: a service must be
registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the
`deploy-remote.sh` dispatch case, its pre-deploy requirement, and
`deploy.sh`'s dispatch regex, where an omission silently plans an empty set
for `--only`; a `dankclaw/<svc>/*` case must precede the `dankclaw/*.container`
glob because that `*` crosses `/`; and the changed-path routing lives *inside*
the per-path loop, after the `scripts/*.env)` case, not at the top of the file.
- Deliberately NOT done: the deploy (pulling 1.27 GiB to Dankclaw was not
asked for), any Caddy vhost or public exposure, audiobook/EPUB (issue #5), any
`frank-tts` code change, and the harbor templates on Dankserver —
`~/.harbor/compose.opennotebook.yml` pins the deprecated `-single` image,
carries no encryption key and injects the whole harbor env — left untouched,
read-only, as the record of what not to copy.
## 2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code
- Third slice of the voice stack (after the façade and the engine decision):
`scripts/build-voice-pack.sh` turns a source recording plus a crop offset into
a deployable pack — `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the
engine's own `reference.spk` (8,192 B) + `reference.rvq` (5,456–6,864 B) made
from it, and `manifest.json` provenance. Tracked tunables in
`scripts/voice-packs.env`: pack list, per-voice source/offset/duration/pinned
source sha256, extractor path and engine commit, artifact arithmetic. New runbook
[[Voice Packs]].
- **The gate is the product.** The project-wide invariant — no voice material
from the first 10 s of a sample file — had only ever been a sentence in
`scripts/frank-tts.env`. An offset inside it now exits 1 before a directory is
created, and the same assertion runs again against the manifest that was just
written, next to the recorded sha256, so a pack cannot carry a forbidden crop
even if the env file is edited into nonsense. `10` stays a constant in the
builder; the env file carries offsets, never the law.
- **The extractor's contract was read, not guessed:** `qwen-codec` in qwentts.cpp
`5916bf7c` takes `--model <codec gguf> --talker <base talker gguf> -i <wav>`
with an optional `--device`, has **no `-o`** and names both artifacts next to
the input with the extension swapped. So the WAV is staged under its final stem
and `reference.wav` — the file the façade gates on — is moved into place
**last**, after the artifacts and the manifest: a failed extractor leaves a
directory that is not a voice. `--force` retires an existing pack whole to
`.AGENTTRASH/` instead of touching bytes; nothing in this script deletes.
- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B per
12.5 Hz frame (`22 × floor(samples/1920)`) and `enc_dim 2048 × f32` = 8,192 B.
That makes the **effective crop window 19.84–25 s**, narrower than the WAV
law's 18–25 s, and a too-short crop is refused before the extractor runs. An
engine bump that changes either number fails the build by design; the bump
procedure is in [[Voice Packs]].
- Verified against reality, not against the model: the pinned `narrator1` crop
(98.739–118.636 s of `audiosample.mp3`, sha256 `2299510…`) comes out of a
`--dry-run` at 955,378 B / 19.897 s with 5,456 B of `.rvq` planned — byte-for-byte
the trial's hand-cut numbers — and all three pinned source digests match the
files on disk.
- Seam: `tests/voice_pack_unit.bats` (13 tests) — refusal inside the forbidden
region creates nothing, a legal crop builds the whole pack, `ffprobe` on the
generated reference, manifest round-trip through `jq` with a sha that matches
the source, re-run-without-`--force` leaves every byte alone, `--force` retires
rather than deletes, `--dry-run` needs no extractor, a wrong-sized artifact
from a bumped engine cannot publish a voice, and every crop in the tracked pack
list clears the invariant. `qwen-codec` is stood in by a stub with the real
tool's I/O shape, so no engine, no GPU, no network, no fleet host. Suite green:
**416/416** (`./tests/run.sh`, wall 77 s) — 403 before this branch.
- Deliberately NOT done: no transcript in the manifest (ICL wants a curated
`ref_text`; the trial's machine transcripts were wrong, and the engine
registers WAV-only voices anyway), no 16 kHz variant (arm A is native 24 kHz),
no loudness pass (not in the pack law), and no pack built on the host — the
three sources still live on the workstation and the engine GGUFs have not been
fetched to `/home/dank/models/Qwen3-TTS-GGUF` yet, so the first real run is a
host step.
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96"><polygon points="48,6 86,27 86,69 48,90 10,69 10,27" fill="#101a12" stroke="#7ee787" stroke-width="3"/><text x="48" y="60" font-family="monospace" font-size="30" font-weight="bold" fill="#7ee787" text-anchor="middle">ON</text></svg>

After

Width:  |  Height:  |  Size: 293 B

+13
View File
@@ -28,6 +28,19 @@
"art_bg": "#f5f5f5", "art_bg": "#f5f5f5",
"warn": "Chats stay on Frank's server, but OpenWebUI's web search rides the same SearXNG path as the search app: obfuscated, not anonymous. The upstream search API keys are tied to Frank’s identity." "warn": "Chats stay on Frank's server, but OpenWebUI's web search rides the same SearXNG path as the search app: obfuscated, not anonymous. The upstream search API keys are tied to Frank’s identity."
}, },
{
"name": "open-notebook",
"category": "user",
"section": "AI",
"description": "Podcast generator — notes in, spoken episode out: script written by frank-zealot, voices rendered by frank-tts. No public route; LAN UI only once it is deployed.",
"url": "",
"docs": "",
"probe": "127.0.0.1:8502",
"method": "http",
"wip": "down",
"showLink": false,
"art": "/img/apps/mono-on.svg"
},
{ {
"name": "Gitea", "name": "Gitea",
"category": "user", "category": "user",
+274
View File
@@ -0,0 +1,274 @@
#!/usr/bin/env bats
# Seam: open-notebook, the fleet's podcast generator on Dankclaw — the Quadlet
# pair (app + SurrealDB), the private network between them, the env file both
# units read, the deployer routing, and the homepage card. Runbook:
# serverAppConfigs-docs/Dankclaw/Open Notebook.md.
#
# Structure only: what the files declare and what the deployer can see. Nothing
# here starts a container, opens a socket, or touches the network — the app's
# behaviour inside the image is upstream's contract, cited in the docs page.
load helpers
UNIT_DIR="dankclaw/open-notebook/quadlet"
APP="$REPO_ROOT/$UNIT_DIR/open-notebook.container"
DB="$REPO_ROOT/$UNIT_DIR/open-notebook-surrealdb.container"
NET="$REPO_ROOT/$UNIT_DIR/open-notebook.network"
ENVF="$REPO_ROOT/scripts/open-notebook.env"
PLANNER="$REPO_ROOT/scripts/deploy-plan.py"
REMOTE="$REPO_ROOT/scripts/deploy-remote.sh"
# The host-side path the units read (they run on Dankclaw, not here), spelled
# out once so the ordering test below compares against a real constant.
HOST_TUNABLES="EnvironmentFile=/home/dank/serverAppConfigs/scripts/open-notebook.env"
HOST_SECRETS="EnvironmentFile=/home/dank/Containers/open-notebook/env"
# The pins are part of the contract, so a silent re-resolution to a different
# build has to fail here rather than at the next deploy. Both digests were
# resolved from the registry on 2026-10-18; the SurrealDB one is 2.7.0.
APP_PIN="sha256:54b2025a4a4853c188f74b9b8246eca8c5480e5dd1babfa92db1038ff2f30212"
DB_PIN="sha256:e5d7387e3727bbe2f7cd6d869e48cd34777272258572ead37c9656b25f745723"
@test "open-notebook ships a network, a database unit and an app unit" {
[ -f "$NET" ]
[ -f "$DB" ]
[ -f "$APP" ]
}
@test "both units are boot-persistent and self-healing" {
for unit in "$APP" "$DB"; do
grep -qE '^Restart=(always|on-failure)$' "$unit" || {
echo "$(basename "$unit") has no Restart="; false; }
# "short RestartSec": a podcast service has no reason to wait minutes.
restart_sec="$(sed -n 's/^RestartSec=\([0-9]*\)s\?$/\1/p' "$unit")"
[ -n "$restart_sec" ] || {
echo "$(basename "$unit") has no RestartSec"; false; }
[ "$restart_sec" -le 30 ] || {
echo "$(basename "$unit") RestartSec=${restart_sec}s is not short"; false; }
grep -qE '^\[Install\]' "$unit"
grep -qE '^WantedBy=default\.target$' "$unit"
done
}
@test "both images are digest-pinned and never re-resolved at start" {
grep -qxF "Image=docker.io/lfnovo/open_notebook:v1-latest@$APP_PIN" "$APP"
grep -qxF "Image=docker.io/surrealdb/surrealdb:v2@$DB_PIN" "$DB"
grep -qxF 'Pull=never' "$APP"
grep -qxF 'Pull=never' "$DB"
# Pull=never is only honest if the fetch it depends on is written down where
# the operator reads the unit, and wired into the deployer's precondition.
grep -qF 'podman pull' "$APP"
grep -qF 'podman pull' "$DB"
grep -qF 'ensure_pinned_image' "$REMOTE"
}
@test "the database port is never published" {
# 8000 belongs to the private bridge. A published 8000 would put the RPC,
# root credentials included, on every interface of the host.
! grep -qE '^PublishPort=' "$DB"
! grep -qE '^PublishPort=.*:8000(:|$)' "$APP"
}
@test "the app publishes its UI and moves the API off the host port seerr holds" {
grep -qxF 'PublishPort=0.0.0.0:8502:8502' "$APP"
# seerr already binds 127.0.0.1:5055 on this host, so the API cannot leave
# the container on the host port upstream uses.
publish="$(grep -E '^PublishPort=[^:]+:[0-9]+:5055$' "$APP")"
[ -n "$publish" ] || { echo "no PublishPort mapping to the API's 5055"; false; }
mapping="${publish#PublishPort=}" # e.g. 0.0.0.0:5056:5055
host_port="${mapping#*:}"
host_port="${host_port%%:*}"
[ "$host_port" != "5055" ] || {
echo "the API is published on host 5055, which seerr already holds"; false; }
# The frontend auto-detects the API as <host it was opened on>:5055, so a
# remap is only correct if API_URL names the published port.
api_url="$(sed -n 's/^API_URL=//p' "$ENVF")"
[ "$api_url" = "http://192.168.0.143:${host_port}" ] || {
echo "API_URL ($api_url) does not name the published API port ($host_port)"; false; }
}
@test "the pair shares one network and the app finds the database by container name" {
grep -qxF 'Network=open-notebook.network' "$DB"
grep -qxF 'Network=open-notebook.network' "$APP"
[ "$(grep -cE '^Network=' "$DB")" = "1" ]
# dankclaw_default is the app's second network and must come FIRST: the
# generated --network flags keep file order, so the default route (and the
# gateway name proven on that network) rides it.
[ "$(grep -cE '^Network=' "$APP")" = "2" ]
[ "$(grep -E '^Network=' "$APP" | head -1)" = "Network=dankclaw_default" ]
db_name="$(sed -n 's/^ContainerName=//p' "$DB")"
[ "$db_name" = "open-notebook-surrealdb" ]
grep -qxF "SURREAL_URL=ws://${db_name}:8000/rpc" "$ENVF"
}
@test "each unit reads the repo tunables first and the host secrets second" {
# podman applies --env-file in argument order and the LAST file wins, so
# swapping these two lines would blank every credential the app needs.
for unit in "$APP" "$DB"; do
files="$(grep -E '^EnvironmentFile=' "$unit")"
[ "$(echo "$files" | wc -l)" = "2" ] || {
echo "$(basename "$unit") must read exactly two env files"; false; }
[ "$(echo "$files" | head -1)" = "$HOST_TUNABLES" ] || {
echo "$(basename "$unit") must read the tracked env file first"; false; }
[ "$(echo "$files" | tail -1)" = "$HOST_SECRETS" ] || {
echo "$(basename "$unit") must read the host secret file last"; false; }
done
}
@test "the interim episode destination is mounted exactly once" {
# The bind that answers Gitea issue #7 provisionally: two of them is a
# shadowing bug, none is episode audio locked inside a named volume.
[ "$(grep -cE '^Volume=/home/dank/media/podcasts:/app/data/podcasts$' "$APP")" = "1" ]
# The app's own state stays in a named volume; the database gets its own.
grep -qE '^Volume=open-notebook-data:/app/data$' "$APP"
grep -qE '^Volume=open-notebook-db:/mydata$' "$DB"
# The deployer creates the host directory as the deploy user.
grep -qF 'mkdir -p /home/dank/media/podcasts' "$REMOTE"
}
@test "the env file declares the CPU-shutdown guard and both timeouts" {
# Upstream defaults are 5 and 5; frank-tts spawns ~8 GiB / 8 threads per
# request on a box that also serves two LLM endpoints.
grep -qxF 'TTS_BATCH_SIZE=1' "$ENVF"
grep -qxF 'OPEN_NOTEBOOK_WORKER_MAX_TASKS=1' "$ENVF"
# Per dialogue line, and strictly beyond frank-tts's own per-request
# ceiling: nested budgets go inside-out (the rule scripts/frank-tts.env
# states for LLAMA_TIMEOUT), so a shim give-up surfaces as frank-tts's
# error instead of the worker's own timeout. The seam reads the ceiling
# rather than restating it, so re-measuring frank-tts cannot leave this
# file silently behind.
tts="$(sed -n 's/^ESPERANTO_TTS_TIMEOUT=//p' "$ENVF")"
shim="$(sed -n 's/^SHIM_TIMEOUT=//p' "$REPO_ROOT/scripts/frank-tts.env")"
[ -n "$tts" ] || { echo "no ESPERANTO_TTS_TIMEOUT in the env file"; false; }
[ -n "$shim" ] || { echo "no SHIM_TIMEOUT in scripts/frank-tts.env"; false; }
[ "$tts" -gt "$shim" ] || {
echo "ESPERANTO_TTS_TIMEOUT=$tts is not beyond frank-tts SHIM_TIMEOUT=$shim"; false; }
# Upstream: keep the scriptwriting call under the web UI's ~600 s patience,
# or the failure arrives as a UI timeout with no name on it.
llm="$(sed -n 's/^ESPERANTO_LLM_TIMEOUT=//p' "$ENVF")"
[[ "$llm" =~ ^[0-9]+$ ]] && [ "$llm" -lt 600 ] || {
echo "ESPERANTO_LLM_TIMEOUT=$llm is not a number under 600"; false; }
}
@test "secrets stay declared-but-empty in the tracked env file" {
# scripts/*.env is un-ignored for "paths and numbers, never credentials":
# the values belong to the host file the deployer seeds.
for key in OPEN_NOTEBOOK_ENCRYPTION_KEY OPEN_NOTEBOOK_PASSWORD SURREAL_PASSWORD SURREAL_PASS; do
grep -qxF "${key}=" "$ENVF" || {
echo "$key is not declared, empty, in $ENVF"; false; }
done
has_no_leaked_secret "$ENVF"
has_no_leaked_sudo_pw "$ENVF"
# And the deployer treats blank as fatal, because blank means "auth off",
# not "unconfigured".
grep -qF 'seed_open_notebook_env' "$REMOTE"
}
@test "the database password never rides in a command line" {
# `surreal start` reads SURREAL_USER / SURREAL_PASS from the environment;
# --pass would put it in /proc/<pid>/cmdline for any ps on the host.
grep -qE '^Exec=start' "$DB"
! grep -qE '^Exec=.*--(pass|user)' "$DB"
}
@test "both health probes are argument-only and name real paths" {
for unit in "$APP" "$DB"; do
cmd="$(sed -n 's/^HealthCmd=//p' "$unit")"
[ -n "$cmd" ] || { echo "$(basename "$unit") has no HealthCmd"; false; }
# The quadlet generator splits HealthCmd on spaces, so anything needing
# a shell (quotes, pipes, &&) would be silently mis-sent.
! printf '%s' "$cmd" | grep -qE '["&;|`]'
done
# The API's own route, exempt from its password middleware upstream.
grep -qxF 'HealthCmd=curl -fsS http://127.0.0.1:5055/health' "$APP"
# The distroless database image has no shell and no curl; it ships a probe.
grep -qxF 'HealthCmd=/surreal is-ready' "$DB"
}
@test "the deployer gates open-notebook on the health path that actually answers" {
grep -qF 'open-notebook) deploy_open_notebook ;;' "$REMOTE"
gate="$(grep -E '^[[:space:]]*wait_for_health open-notebook ' "$REMOTE" | head -1)"
[ -n "$gate" ] || { echo "deploy_open_notebook has no wait_for_health gate"; false; }
# /health is the route upstream serves; the suite must not bless a guess.
printf '%s' "$gate" | grep -qE '/health( |$)'
# The gate is a host-loopback URL (the publish is wildcard, so 127.0.0.1
# answers) on the PUBLISHED port, never the container's 5055.
printf '%s' "$gate" | grep -qE 'http://127\.0\.0\.1:[0-9]+/health'
! printf '%s' "$gate" | grep -qE ':5055/'
# The database is gated by its container health before the app is started.
grep -qE 'wait_container_healthy open-notebook-surrealdb' "$REMOTE"
}
@test "the deployer refuses to start the pair on blank secrets and never rewrites a seeded file" {
# Read the seed function itself: the refusal, and the no-clobber rule that
# protects a password an operator set by hand.
seed="$(sed -n '/^seed_open_notebook_env()/,/^}/p' "$REMOTE")"
[ -n "$seed" ]
printf '%s' "$seed" | grep -qF 'return 1'
printf '%s' "$seed" | grep -qF 'if [[ ! -e "$dest" ]]'
printf '%s' "$seed" | grep -qF 'chmod 600'
# One value, two spellings: the app reads SURREAL_PASSWORD, the CLI SURREAL_PASS.
printf '%s' "$seed" | grep -qF 'SURREAL_PASS='
}
@test "the planner maps every open-notebook path to the open-notebook service" {
run python3 "$PLANNER" --target dankclaw --paths \
dankclaw/open-notebook/quadlet/open-notebook.container \
dankclaw/open-notebook/quadlet/open-notebook-surrealdb.container \
dankclaw/open-notebook/quadlet/open-notebook.network \
scripts/open-notebook.env
[ "$status" -eq 0 ]
[ "$output" = "open-notebook" ]
# One deployable owns the pair: the unit files never plan separately.
run python3 "$PLANNER" --target dankclaw --paths \
dankclaw/open-notebook/quadlet/open-notebook-surrealdb.container
[ "$output" = "open-notebook" ]
# And the service belongs to dankclaw only.
run python3 "$PLANNER" --target dankserver --paths scripts/open-notebook.env
[ "$status" -eq 0 ]
[ -z "$output" ]
}
@test "changed-path routing selects this seam" {
run "$REPO_ROOT/scripts/test-changed.sh" --list --paths \
dankclaw/open-notebook/quadlet/open-notebook.container
[ "$status" -eq 0 ]
printf '%s\n' "$output" | grep -qx "tests/open_notebook.bats"
run "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/open-notebook.env
[ "$status" -eq 0 ]
printf '%s\n' "$output" | grep -qx "tests/open_notebook.bats"
}
@test "both deployer entry points know the service" {
grep -qF 'dankclaw:open-notebook|' "$REMOTE"
grep -qF 'dankclaw:open-notebook|' "$REPO_ROOT/scripts/deploy.sh"
}
@test "the homepage card exists and stays hidden until the service is deployed" {
python3 - "$REPO_ROOT/shared/homepage/services.json" <<'PY'
import json, os, sys
data = json.load(open(sys.argv[1]))
cards = [s for s in data if s["name"] == "open-notebook"]
assert len(cards) == 1, f"expected exactly one open-notebook card, got {len(cards)}"
card = cards[0]
assert card["category"] == "user" and card["section"] == "AI", card
assert card["probe"] == "127.0.0.1:8502", card["probe"]
assert card["method"] == "http"
# Not deployed yet: the card is reported wip-down and carries no link, so the
# page shows no dead button (the Seerr precedent).
assert card.get("wip") == "down", card.get("wip")
assert card.get("showLink") is False, card.get("showLink")
assert not card.get("url"), "a hidden card must not carry a url"
art = card["art"].lstrip("/")
assert os.path.exists(os.path.join(os.path.dirname(sys.argv[1]), art)), art
PY
}
@test "the runbook page exists and names the interim mount's owning issue" {
page="$REPO_ROOT/serverAppConfigs-docs/Dankclaw/Open Notebook.md"
[ -f "$page" ]
grep -qF 'issue #7' "$page"
grep -qF 'human-only' "$page"
# The interim decision is one mount line; the page has to say how to undo it.
grep -qF '## Rollback' "$page"
}
+268
View File
@@ -0,0 +1,268 @@
#!/usr/bin/env bats
# Seam: the voice-pack builder (scripts/build-voice-pack.sh).
#
# The contract this pins is the one frank-tts's packs live or die by. A crop is
# only usable voice material if it starts PAST the first 10 seconds of the
# source file — the project-wide source-material invariant — and this builder is
# the thing that enforces it, because nothing downstream looks at offsets again.
# A pack is three files plus provenance: reference.wav at 24 kHz mono inside the
# 18-25 s window, the engine's own speaker artifacts beside it, and a
# manifest.json whose sha256 still matches the bytes the crop came from. Builds
# refuse to clobber what is already on disk, and --force retires rather than
# deletes.
#
# No engine binary, no model, no GPU, no network and no fleet host: the crop is
# cut from a locally generated tone and the pinned extractor is stood in by a
# stub that mirrors the real tool's I/O shape (no -o flag: outputs named next to
# the input with the extension swapped, at the sizes the codec's config
# implies), which is how frank_tts_unit.bats drives its backends too.
load helpers
setup() {
BUILDER="$REPO_ROOT/scripts/build-voice-pack.sh"
ENV_FILE="$REPO_ROOT/scripts/voice-packs.env"
TEST_TMP="$(mktemp -d)"
PACKS="$TEST_TMP/packs"
mkdir -p "$PACKS"
STUB_ARGV="$TEST_TMP/stub-argv.tsv"
export STUB_ARGV
# Forty seconds of "recording": stereo 44.1 kHz, so a legal build has to
# downmix and resample just as it does on a real audiobook. Every crop this
# file takes starts at or past 12 s, well out of the forbidden region.
ffmpeg -v error -y -f lavfi -i "sine=frequency=330:sample_rate=44100:duration=40" \
-ac 2 -c:a pcm_s16le "$TEST_TMP/source.wav"
SOURCE_SHA="$(sha256sum "$TEST_TMP/source.wav" | awk '{ print $1 }')"
write_stub_tool
# The shipped tunables file, then the three machine-local overrides that
# point the engine at the stub: the real env stays under test.
cat "$ENV_FILE" > "$TEST_TMP/env"
{
printf 'VOICE_PACK_DIR=%s\n' "$PACKS"
printf 'VOICE_PACK_TOOL=%s\n' "$TEST_TMP/qwen-codec"
printf 'VOICE_PACK_CODEC_MODEL=%s\n' "$TEST_TMP/codec.gguf"
printf 'VOICE_PACK_TALKER_MODEL=%s\n' "$TEST_TMP/talker.gguf"
} >> "$TEST_TMP/env"
printf 'stub-codec-weights\n' > "$TEST_TMP/codec.gguf"
printf 'stub-talker-weights\n' > "$TEST_TMP/talker.gguf"
}
teardown() {
rm -rf "$TEST_TMP"
}
# write_stub_tool — stand-in for qwen-codec. Records what it was handed and
# writes <input-stem>.spk / <input-stem>.rvq, exactly where the real tool would.
# STUB_SPK_BYTES lets one test be an engine that changed its x-vector size.
write_stub_tool() {
cat > "$TEST_TMP/qwen-codec" <<'STUB'
#!/usr/bin/env bash
set -euo pipefail
input="" model="" talker=""
while [ "$#" -gt 0 ]; do
case "$1" in
--model) model="$2"; shift 2 ;;
--talker) talker="$2"; shift 2 ;;
--device) shift 2 ;;
-i) input="$2"; shift 2 ;;
*) shift ;;
esac
done
printf 'model=%s\ttalker=%s\tinput=%s\n' "$model" "$talker" "$input" >> "$STUB_ARGV"
[ -n "$input" ] || exit 2
[ -s "$model" ] || exit 3
[ -s "$talker" ] || exit 4
dur="$(ffprobe -v error -select_streams a:0 -show_entries stream=duration -of default=nw=1:nk=1 "$input")"
frames="$(awk -v d="$dur" 'BEGIN { printf "%d\n", int(int(d * 24000 + 0.5) / 1920) }')"
head -c "${STUB_SPK_BYTES:-8192}" /dev/zero > "${input%.wav}.spk"
head -c "$((22 * frames))" /dev/zero > "${input%.wav}.rvq"
printf '[Stub] Wrote %s.rvq: %d packed bytes\n' "${input%.wav}" "$((22 * frames))" >&2
STUB
chmod +x "$TEST_TMP/qwen-codec"
}
# build — a real (non-dry) run against the stub tool with the tone as source.
build() {
local voice="$1"
shift
run "$BUILDER" --env "$TEST_TMP/env" \
--source "$TEST_TMP/source.wav" --sha256 "$SOURCE_SHA" "$@" "$voice" "$PACKS"
}
pack_files() {
ls "$PACKS/$1" 2>/dev/null | sort | tr '\n' ' '
}
@test "a crop offset inside the first ten seconds is refused and names the invariant" {
build narrator1 --offset 4.5 --duration 20
[ "$status" -ne 0 ] || { echo "$output"; false; }
[[ "$output" == *"source-material invariant"* ]]
[[ "$output" == *"first 10 s"* ]]
# Refused before anything was created, not cleaned up afterwards.
[ ! -e "$PACKS/narrator1" ]
[ ! -s "$STUB_ARGV" ]
}
@test "the identical crop at a legal offset builds the whole pack" {
build narrator1 --offset 12 --duration 20
[ "$status" -eq 0 ] || { echo "$output"; false; }
[[ "$output" == *"CROP 12 s .. 32 s"* ]]
[[ "$output" == *"VOICE_PACK_BUILT narrator1 $PACKS/narrator1"* ]]
[ -f "$PACKS/narrator1/reference.wav" ]
[ -f "$PACKS/narrator1/reference.spk" ]
[ -f "$PACKS/narrator1/reference.rvq" ]
[ -f "$PACKS/narrator1/manifest.json" ]
}
@test "reference.wav comes out 24 kHz mono pcm inside the 18-25 s window" {
build narrator1 --offset 12 --duration 20
[ "$status" -eq 0 ] || { echo "$output"; false; }
local wav="$PACKS/narrator1/reference.wav"
[ "$(ffprobe -v error -select_streams a:0 -show_entries stream=codec_name -of default=nw=1:nk=1 "$wav")" = "pcm_s16le" ]
[ "$(ffprobe -v error -select_streams a:0 -show_entries stream=sample_rate -of default=nw=1:nk=1 "$wav")" = "24000" ]
[ "$(ffprobe -v error -select_streams a:0 -show_entries stream=channels -of default=nw=1:nk=1 "$wav")" = "1" ]
local dur
dur="$(ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 "$wav")"
awk -v d="$dur" 'BEGIN { exit !(d >= 18 && d <= 25) }'
}
@test "the manifest round-trips through jq with offsets and a sha that matches the source" {
build narrator2 --offset 15 --duration 22.5
[ "$status" -eq 0 ] || { echo "$output"; false; }
local manifest="$PACKS/narrator2/manifest.json"
jq -e '
.voice == "narrator2"
and (.crop.offset_seconds == 15)
and (.crop.end_seconds == 37.5)
and (.reference.sample_rate == 24000)
and (.reference.channels == 1)
and (.reference.duration_seconds >= 18 and .reference.duration_seconds <= 25)
and (.artifacts.engine_commit | test("^[0-9a-f]{40}$"))
and (.artifacts.produced == true)
and (.invariant.forbidden_source_seconds == 10)
' "$manifest"
# The recorded hash is the source's hash, not a hash of something else.
[ "$(jq -r .source.sha256 "$manifest")" = "$(sha256sum "$TEST_TMP/source.wav" | awk '{ print $1 }')" ]
# The pack's provenance names the tool that made the artifacts and the files
# it wrote beside the WAV.
[ "$(jq -r .artifacts.tool "$manifest")" = "$TEST_TMP/qwen-codec" ]
grep -q "input=$PACKS/narrator2/staging-reference.wav" "$STUB_ARGV"
}
@test "a wrong pinned sha stops the build before a crop is cut" {
run "$BUILDER" --env "$TEST_TMP/env" --source "$TEST_TMP/source.wav" \
--sha256 "$(printf '0%.0s' {1..64})" --offset 12 --duration 20 narrator1 "$PACKS"
[ "$status" -ne 0 ]
[[ "$output" == *"sha256"* ]]
[ ! -e "$PACKS/narrator1" ]
}
@test "a crop that runs off the end of the source is refused" {
build narrator1 --offset 30 --duration 20
[ "$status" -ne 0 ]
[[ "$output" == *"crop would come out short"* ]]
[ ! -e "$PACKS/narrator1" ]
}
@test "a crop under the engine's artifact window is refused before the extractor runs" {
# 18.5 s is inside the reference window but packs to 22 * 231 = 5,082 B of
# .rvq, under the 5,456 B the pinned codec config produces — the builder
# says so rather than running an extractor that would emit it anyway.
build narrator1 --offset 12 --duration 18.5
[ "$status" -ne 0 ] || { echo "$output"; false; }
[[ "$output" == *"5456"* ]]
[ ! -e "$PACKS/narrator1" ]
[ ! -s "$STUB_ARGV" ]
}
@test "an extractor whose artifact size changed does not get a published voice" {
# The engine-bump case: a 4,096 B x-vector is not the 8,192 B the pinned
# encoder emits, so the pack must not reach the state the shim reads.
export STUB_SPK_BYTES=4096
build narrator1 --offset 12 --duration 20
[ "$status" -ne 0 ]
[[ "$output" == *"reference.spk"* ]]
[ ! -e "$PACKS/narrator1/reference.wav" ]
[ ! -e "$PACKS/narrator1/manifest.json" ]
}
@test "re-running without --force refuses and leaves every byte alone" {
build narrator1 --offset 12 --duration 20
[ "$status" -eq 0 ] || { echo "$output"; false; }
sha256sum "$PACKS"/narrator1/* > "$TEST_TMP/before.sha"
build narrator1 --offset 12 --duration 20
[ "$status" -ne 0 ] || { echo "$output"; false; }
[[ "$output" == *"--force"* ]]
sha256sum -c --quiet "$TEST_TMP/before.sha"
[ "$(find "$PACKS" -name .AGENTTRASH -prune -o -type f -newer "$TEST_TMP/before.sha" -print | wc -l)" -eq 0 ]
}
@test "--force retires the previous pack instead of deleting anything" {
build narrator1 --offset 12 --duration 20
[ "$status" -eq 0 ] || { echo "$output"; false; }
cp "$PACKS/narrator1/reference.wav" "$TEST_TMP/first-reference.wav"
build narrator1 --offset 12 --duration 20 --force
[ "$status" -eq 0 ] || { echo "$output"; false; }
[[ "$output" == *"RETIRED"* ]]
# The old bytes survive, whole, under the trash directory beside the pack.
local retired
retired="$(find "$PACKS/.AGENTTRASH" -name reference.wav | head -n1)"
[ -n "$retired" ]
cmp "$retired" "$TEST_TMP/first-reference.wav"
[ -f "$PACKS/narrator1/reference.wav" ]
}
@test "--dry-run needs no extractor at all" {
# Everything up to the engine: the gate, the crop, the probe, the plan and
# the manifest it would write — with the tool path pointed at nothing.
run "$BUILDER" --env "$TEST_TMP/env" --tool "$TEST_TMP/no-such-tool" \
--source "$TEST_TMP/source.wav" --sha256 "$SOURCE_SHA" \
--offset 12 --duration 20 --dry-run narrator1 "$PACKS"
[ "$status" -eq 0 ] || { echo "$output"; false; }
[[ "$output" == *"not present"* ]]
[ ! -e "$PACKS/narrator1" ]
[ ! -s "$STUB_ARGV" ]
printf '%s\n' "$output" | awk '/^MANIFEST_BEGIN$/{p=1;next} /^MANIFEST_END$/{p=0} p' \
| jq -e '.voice == "narrator1" and .crop.offset_seconds == 12 and .artifacts.produced == false'
}
@test "a name the façade could not resolve is refused" {
# app.py matches the directory name verbatim while the engine lowercases
# names, and a slash would escape $VOICES_DIR: both are refused here, at the
# only moment anyone can still fix them.
for bad in 'Narrator1' '../evil' 'voice name'; do
run "$BUILDER" --env "$TEST_TMP/env" --source "$TEST_TMP/source.wav" \
--offset 12 --duration 20 "$bad" "$PACKS"
[ "$status" -ne 0 ] || { echo "accepted '$bad': $output"; false; }
[[ "$output" == *"voice name"* ]]
done
[ ! -e "$PACKS/Narrator1" ]
[ ! -e "$PACKS/evil" ]
}
@test "every crop in the tracked pack list clears the invariant" {
# The law applied to the values the repo actually ships: no pack may be
# described by an offset inside the forbidden region, a length outside the
# reference window, or a source with no digest to verify it against.
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
local checked=0 pack upper
read -r -a packs <<<"$VOICE_PACKS"
for pack in "${packs[@]}"; do
upper="$(printf '%s' "$pack" | tr '[:lower:]' '[:upper:]')"
local offset_var="VOICE_PACK_${upper}_OFFSET" duration_var="VOICE_PACK_${upper}_DURATION"
local sha_var="VOICE_PACK_${upper}_SHA256" source_var="VOICE_PACK_${upper}_SOURCE"
[ -n "${!source_var:-}" ] || { echo "$pack has no source path"; false; }
printf '%s' "${!sha_var}" | grep -Eq '^[0-9a-f]{64}$' || { echo "$pack has no pinned sha256"; false; }
awk -v o="${!offset_var}" 'BEGIN { exit !(o >= 10) }' || { echo "$pack crops at ${!offset_var} s, inside the forbidden first 10 s"; false; }
awk -v d="${!duration_var}" 'BEGIN { exit !(d >= 18 && d <= 25) }' || { echo "$pack crops ${!duration_var} s, outside 18-25 s"; false; }
checked=$((checked + 1))
done
[ "$checked" -gt 0 ]
}