18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).
Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.
Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV
accounting, MTP verdict, compaction-hub concurrency), and two Key
Lessons bullets -- the draft-context bill restated with the real knob
names, plus a new lesson that a quant's scale overhead belongs in its
byte rate (the 6.25 % invented by doubling 13,056.00 MiB).
LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape;
the 3-slot q8_0 boot ledger and its measurements are relabelled as the
arm this replaced (byte-identical numbers, honest attribution), the
corrected f16 rate derivation is added, the MTP census is labelled
q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the
boot-ledger section carries an explicit PROJECTION that must be replaced
with the measured ledger from the first f16 boot.
Pointers re-pointed (live-shape sentences only; every historical ledger
left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md,
and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k,
qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
A consumer that lists the 29-name af_*/am_*/bf_*/bm_* pack and refuses
narrator1 without a request reaching the façade is synthesizing in the
browser. Frank TTS's troubleshooting table gains that symptom; OpenWebUI's
See Also gains the service page.
The `Voice "narrator1" not found. Should be one of: af_heart, …, bm_fable`
toast is OpenWebUI 0.11.4's bundled Kokoro.js validator: the enum is
Kokoro's fixed 29-voice pack, not frank-tts's registry. Frank-tts answers
200 audio/mpeg for the exact OpenWebUI body from inside the container, and
the failing click leaves no `audio/speech` line in the container log — that
contrast is the instrument. Admin config DB is correct
(engine=openai, base URL :7772/v1, model frank-tts, voice narrator1); no
container env and no user row carries an audio override, so the
`browser-kokoro` choice is per-browser storage. Fix: Settings → Audio →
TTS Engine → Default.
#18 shipped DEFAULT_VOICE and #17 explained why the harness cannot name a voice from a setting, but the endpoint page still described voice resolution as if every request carried a name. One contract row, one paragraph in Naming a voice (a call that names nothing is the harness default shape, not a client bug), the env-file row now lists the default voice among its knobs, and the worklog records the rollout: both merges, the one-dispatch deploy with the 33/33 gate, the card flipping to up on the next checker tick, and the previously failing call returning 430124 bytes with the journal reading "speech voice=narrator1 (default)".
The section this PR adds claimed speech.voice was the forwarded voice key. It is not: the openai-speech transport builds {model, input, response_format, voice?} and sets voice only when the CALL passed voice_id, tts.localVoice is the local Kokoro backend key, and speech.voice is the TUI speak-aloud voice whose enum is Kokoro-only. A restarted session proved it the honest way - the tts device arrived at 7772 with no voice key and got "unknown voice None; available voices: narrator1, narrator2, narrator3" - so the fresh-process success earlier that day was the run that was told to pass voice_id, not evidence of a setting.
The law now reads: voice comes from voice_id alone, so a voice-less call is the harness default shape (the endpoint grew DEFAULT_VOICE in its own PR to answer it), and speech.voice must stay a Kokoro name - a pack name there breaks the vocalizer and does nothing for synthesis. config.yml reverted to am_fenrir on the operator box, and the worklog entry that recorded narrator1 as wiring carries the correction.
Six statements still claimed the endpoint did not exist: both vault descriptions, the two index.md bullets, the Open Notebook blocker, and the engine page deploy order that opened with "Nothing below has been run". All re-keyed to the deployed state, with the ledger the deploys printed: engine healthy 31 s after the build at 4.0 GiB resident, three voices warm, the ladder measured at 1883 ms then 1226-1283 ms (not the retired 6.2-8.0 s), and two voices rendering the same sentence into different bytes.
Adds a The harness speech role section to the endpoint page with the three wiring laws measured the day of the flip: speech.voice is the forwarded key (tts.localVoice belongs to Kokoro), an explicit speech role has no fallback candidate, and models.yml loads at process start while config.yml is read live - so a session older than the frank-tts row fails exactly like a dead endpoint. Fresh-process proof: omp -p --tools tts saved 180524 bytes with voice=narrator1 while the facade logged the matching speech line.
PR #15 corrected the engine's page, the research brief's law bullet and the
warmer's header, but the retired ladder had six more homes: both quadlet files
("6.2-8.0 s… 0.5 ms", "~6-8 s each", "paying the 5.5 s cold decode"),
`scripts/frank-tts-engine.env` (the whole law paragraph and the WARM_TEXT note),
`scripts/frank-tts.env`, the [[Frank TTS]] troubleshooting row that told an
operator to expect ~5.5 s, the research brief's consequences paragraph, and one
test comment.
All re-keyed to what was measured on the deployed container 2026-10-18: the
first synthesis of a freshly registered voice costs 1,883 ms against 1,226-1,283
ms afterwards, so the warm pass buys about 0.6 s — and the reason it exists at
all, which the pages never led with, is that the registry is memory-only: without
the hook every voice name is a 400 until somebody synthesizes. The trial's dated
worklog entry stays as written; the correction points at it.
No behavior change: comments, unit lore and docs only, so no cfhash move and no
image rebuild.
The first real boot gave the chance to measure what warming actually buys, and
the trial's numbers were wrong:
- Empty registry, register only, then time: the FIRST synthesis of a freshly
registered voice cost 1 883 ms; the next two 1 283 and 1 226 ms. A ~110-word
request cost 9.4 s and then 9.8 s.
- So the published 6.2–8.0 s cold / first-audio-byte 5 530 ms / later-requests
0.5 ms ladder does not reproduce, and request time tracks the audio generated
rather than snapshot state. The 0.5 ms figure is an internal state-restore
cost, never an end-to-end request time.
- Warming still earns its place for a different reason than the page claimed:
the registry is memory-only and re-registering does not warm, so a boot without
the hook leaves the façade answering 400 for every name. The saving is ~0.6 s.
Law 1 of [[Frank TTS Engine]] is re-titled the registry law and carries the
measured numbers; its example journal line is a real one from today; the
troubleshooting row for "first request of the day is slow" — the retired law's
symptom — is replaced by the argument-cap fault that actually happened. The same
ladder in the research brief is annotated rather than erased, and [[Voice Packs]]
step 6 stops claiming the hook registers only packs it has not seen (the engine
replaces a name it already holds).
Three pages, all from faults found this session:
- `LLM/deployment.md` gains "Locally built images": which images have no
registry digest, what the `cfhash` skip is for, why it must cover every COPY'd
file (the silent shape: build skipped, /health green, old code serving), and
the base-image half of the law — the engine's `useradd`, because the pinned
ubuntu:24.04 has no `adduser`.
- `Dankclaw/Frank TTS Engine.md` troubleshooting gains the two lines an operator
actually needs: every voice cold on a fresh boot with a hand-run hook
succeeding (= `WARM_TEXT` arrived quoted, `podman exec … env | grep WARM_TEXT`
proves it), and a 502 from the façade beside a 200 `/health` from the engine
(model loaded ≠ voices registered). The stale-warmer row is re-keyed to the
new rebuild rule.
- `Dankclaw/Voice Packs.md` step 4 was wrong: it told the next agent to restart
the engine *instead of* the warm hook because the hook "only registers packs it
has not seen". It registers every pack on every run and the engine replaces a
name it already holds — `tools/tts-server.cpp:148` at the pinned commit:
"Re-registering a name replaces the previous entry." A restart is what an
engine bump needs (the registry is memory-only), never worse, only slower.
- `Dankclaw/Frank TTS.md` deploy command re-keyed to `./scripts/deploy.sh
dankclaw --service frank-tts`, the way it is actually deployed; the raw
`deploy-remote.sh` form is plumbing.
Stage 2 of the voice stack. Repo state only: nothing deployed, no host touched,
no container started.
- `frank-tts` is now registered everywhere a dankclaw service has to be:
`SERVICE_ORDER` (after `frank-tts-engine`, before the consumers), the planner's
dankclaw set, `deploy-remote.sh`'s allowlist + dispatch + a
`deploy_frank_tts` handler, and `deploy.sh`'s dispatch regex — plus the
`build_compose_image` arm naming `dankclaw/voiceservice/Containerfile`, without
which the build silently returns 0 and the unit's `Pull=never` tag never
exists.
- The unit joins TWO networks, in order: `dankclaw_default` first (default route,
where consumers and open-notebook live), then the voice bridge. `TTS_UPSTREAM`
moves from `host.containers.internal:8880` to `http://frank-tts-engine:8880` —
the actual fix for a shape that could never work, since a bridge container
cannot reach a host-loopback bind and the engine publishes nothing at all.
- `Wants=`/`After=` on the engine, never `Requires=`: the `llama-tts-cli` rollback
backend needs no upstream, so a broken engine may delay this endpoint but must
not take its unit down.
- The handler installs the engine's quadlet pair too, because a referenced
`Network=` file the generator cannot find means no façade unit is produced at
all, and `--only frank-tts` has no other way to get it. It creates
`/home/dank/voices` as the deploy user (rootless podman would auto-create a
missing bind source root-owned, which reads as "no voices"), and gates on
podman's recorded health.
- Voice naming is now written down where it can be found: a voice is a lowercase
directory name matching `^[a-z0-9][a-z0-9._-]*$`, at most 64 chars. That shape
is the intersection of two rules that disagree — the shim would accept mixed
case, the engine lowercases names — so a mixed-case pack resolves on one side
only, and the boot warmer refuses it loudly. `narrator1/2/3` are load-bearing
for open-notebook.
- Homepage card for the pair, `wip: down` and hidden, probing `127.0.0.1:7772`;
the engine gets no card because it publishes nothing and could never be probed.
- **No Prometheus job, deliberately.** No `/metrics` exists on the shim, this
Prometheus 3 refuses HTML/JSON bodies (the existing `comfyui` and `services`
probes are documented down for exactly that), and `LlmEndpointDown` is scoped
`up{job=~"frank-.*"}` — so a `/health` job would page on Signal for a healthy
endpoint. Recorded in the page instead of quietly skipped.
- Vault: new [[Frank TTS]] page linked from the index; [[Frank TTS Engine]] now
points at it for the pair's public signal; log entry for the day.
- Seam grows to 25 tests: the generator's rendering of the pair (both
attachments, first-network order, one publish, Wants-not-Requires, bounded
per-unit blocks), the env file's upstream checked as a network JOIN rather than
a string, planner separation and order, the handler's steps, the card's gating,
routing that selects both seams for either deployable.
Repo state only: nothing deployed, no host touched, no container started.
- `frank-tts-engine` on Dankclaw: `qwentts.cpp` commit
5916bf7c79dbb4e66fc059cdc507a79a23a28bb4, resident `tts-server`, 1.7 B base
talker Q4_K_M + Q8_0 tokenizer, container port 8880. It is deliberately NOT a
fleet endpoint — `frank-tts` on 7772 is the endpoint consumers address.
- Publishes NOTHING. A host-loopback publish would have been useless to the
façade (netavark DNATs published ports only, so a bridge container cannot
reach a host loopback bind — the shape that 502'd hermes) and would have been
the one LAN-reachable surface of an unauthenticated synthesiser. Both voice
containers sit on one committed quadlet network, `frank-tts-engine.network`
pinned to 10.89.5.0/24 (next free /24 after open-notebook's 10.89.4.0/24), and
peers address the engine by container name. Diagnostics and the deployer's
health gate run inside the netns (`podman exec`, `wait_container_healthy`).
- Cold-voice law honoured in the unit: the first request for a registered voice
decodes its whole reference to snapshot codec state (6.2-8.0 s, first audio
byte 5 530 ms), later ones restore in 0.5 ms, and the registry is memory-only.
`ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh`
with `TimeoutStartSec=900`; the `-` prefix keeps a failed warm from
crash-looping the endpoint under `Restart=always`. It logs cold=/warm= per
voice, so the under-a-second second request is proven on every boot.
- Thread law recorded: `tts-server` has no `--threads`; src/backend.h uses
hardware_concurrency()/2, so a cpuset pin silently changes the thread count.
- Image builds from the pinned commit with the proven line (GGML_BLAS=OFF
forced by this host, every GPU backend off, -j 8, no -DGGML_NATIVE because
ggml defaults to -march=native for the building host). Two stages; the runtime
stage COPYs both the entrypoint and the warmer, because a container that
publishes nothing is reached with `podman exec`, not a port.
- `scripts/download-frank-tts-engine-qwentts.sh` fetches the two GGUFs with byte
and LFS-oid checks (talker 1 219 245 248 B, tokenizer 291 150 624 B); the
deployer gates on --verify-only instead of pulling 1.4 GiB mid-deploy.
- Registered in SERVICE_ORDER, DANKCLAW_SERVICES, the deploy-remote.sh dispatch,
its pre-deploy requirement, deploy.sh's regex, the planner's path map (the
whole `dankclaw/voiceservice/` tree mapped to nothing before this), and
test-changed.sh ABOVE the `dankclaw/*.container` case whose `*` crosses `/`.
- New seam tests/frank_tts_engine.bats (19): the Quadlet generator's own output
for network/publish/boot/warm shape, entrypoint argv driven from the tracked
env file, the warmer against a stub tts-server over a real socket
(registration keys, ref_text absent, late engine, hung synthesis, name
refusal), the fetcher's verifier, the pinned cmake line, and both routing
tables. Suite: 422 passed.
- Vault: [[Frank TTS Engine]] linked from the index; log.md entry for the day.
The page a new operator needs: what the three required files are and who
reads each one, how a pack is built (with the extractor's usage text
verbatim, because the flags are not guessable and there is no -o), the
10-second gate and where it fires twice, how to add a fourth voice
including the warm hook that registers a pack without restarting the
engine, and the engine-bump procedure for when the artifact sizes move.
Plus the host census as of 2026-10-18 — the trial packs carry no
artifacts and their manifest is the trial's own schema, so the builder's
first real run is a --force migration — and the three gaps left open on
purpose: no curated ICL transcript, no 16 kHz variant, no loudness pass.
Linked from the Dankclaw services list; log entry appended oldest-first.
The env file first landed with ESPERANTO_TTS_TIMEOUT equal to SHIM_TIMEOUT=600.
Equality is a race: both budgets expire at the same instant, so the case the
number existed to prevent — the worker abandoning a synthesis frank-tts is still
finishing — is exactly what happens on the tail of a long line. Nested budgets
go inside-out, which is the rule scripts/frank-tts.env states for its own
LLAMA_TIMEOUT; the client waits one round-trip longer than the server, so a
frank-tts give-up arrives labelled as frank-tts's error. 660.
The seam's assertion moves from equality to `>` for the same reason, and now
also holds ESPERANTO_LLM_TIMEOUT under upstream's 600 s web-UI patience — above
it the failure surfaces as a UI timeout with no name on it. The scriptwriting
number itself stays at the app's 180 s default: frank-zealot answers an outline
or a transcript well inside it, which corrects the earlier commit message's
claim that both timeouts were raised past upstream's 300 s.
Page registered in the vault index (the docs-completeness seam fails an
unregistered page) and logged in the bundle change log.
What the page owes the reader: the provider contract (frank-zealot for scripts
with a /v1 base URL, frank-tts for voices with 1-4 speakers mapped to the
narrator1/2/3 voice packs typed as free text), why Test Connection and Discover
Models fail cosmetically against frank-tts rather than being a fault, the
encryption key as the precondition for saving any credential, and the honest
blocker — frank-tts is not deployed and nothing listens on 7772, so a generation
run writes its script and fails on the first voice.
Episode destination is stated as interim and cites Gitea issue #7 (human-only,
agents must not work it): the app keeps its named volume, one bind mount puts
finished MP3s under /home/dank/media/podcasts, and deleting that one line is the
whole rollback. Rollback section also covers the full undo, including that the
podman network only disappears once the units are gone.
The cutover deploy's FIRST boot was not the shipped payload: a stale
frank-zealot.service.d/ptq1.conf from the bench window still pointed
FRANK_ZEALOT_ENV_FILE at /tmp/zealot-arm-current.env, so the unit booted the
unmeasured 27B PTQ1_0 arm (3 slots of 133,888, 21,188 MiB) while /health
answered 200, /v1/models echoed the alias and the e2e gate passed 33/33.
Drop-in retired to ~/.AGENTTRASH/20261018-zealot-ptq1-arm/; the 13:19 restart
is the shape the docs now quote (n_ctx_slot 262144, KV 13,056.00 MiB,
settled 17,922/6,203).
Corrections the production ledger forced:
- RS buffer is 150.75 MiB (3 cells) on the shipped spec-off shape; 452.25 MiB
belongs to the SPEC-ARMED arms, where MTP depth 2 bills three seqs per slot.
The drafter triples the recurrent state on top of its 3,072 MiB of f16 KV.
- Decode measured 168.4-168.7 t/s single-stream and ~300 t/s aggregate over
three streams (102.8 each). Wall-clock timing of a generation that stops on
its own reads 4x low (72 t/s for a 24-token countdown) - measure with
ignore_eos and read the engine's timings.
- OpenWebUI stores both decoration values inside JSON columns (params/meta),
and frank-dragoon's context.length had drifted to 262,144 against its own
200,704 ceiling; both re-keyed in the live DB.
New law in AGENTS.md: after deploying a recently-benched LLM service, check
DropInPaths, then /proc/<pid>/cmdline, then the ledger. Health checks cannot
see the payload.
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.
Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
- serve script: header + every fallback default re-keyed to the Bonsai-2 trio
(vendor LLAMA_BIN, PQ2_0 MODEL_FILE, Q8_0 MMPROJ) and CACHE_TYPE_*:=q8_0 —
a defaulted run must land on the shipped shape, never on the retired 9B/f16
pair (the trial window proved a silent default is how a swap goes wrong)
- unit Description now names the payload + KV dtype that `systemctl status`
will show during the next incident
- OpenWebUI decoration-row description re-keyed, with the law that the row is
live DB state a swap must follow into the database
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
- AGENTS.md Model Config: the Protoss names are now the ids on the wire
(per-endpoint knob named); verify a rename on /v1/models, never on a chat
response — radiance echoes the requested name
- OMP Harness.md: harness id == served id; one id per model everywhere
- Homepage.md: the docs-page "api id" annotations and the switchover promise
are retired
- OpenWebUI.md: the id-keyed trio (model row, access_grant, api_configs
model_ids) is a foreign-key set — re-key it in the same deploy; plus the
contenteditable-composer probe note
- frank-secondary/frank-large/frank-primary runbooks: served ids
frank-zealot/frank-dragoon/frank-archon, and the bench.conf drop-in that
made every frank-primary deploy since 2026-10-08 04:09 restart the staging
worktree behind a green health gate
- style.css: green phosphor -> amber CRT-AMBIOS palette; docs styles for
screenshots, code blocks, key warning (full + bare mode)
- index/admin/docs: VT323 from the Google Fonts CDN
- docs.html: rewrite #chat/#models for the live fleet (flash-server v2,
radiance native FP8, Qwen3.5-9B Q6_K); new #api-access with OpenWebUI
key walkthrough + Opencode and Pi configs; three OpenWebUI screenshots
- services.json: AI card details match the current fleet
- deploy-remote.sh: ship shared/homepage/img/ to /srv/homepage/
- docs: Homepage.md theme wave + OpenWebUI.md picker access-control law
OpenWebUI model picker narrowed to exactly frank-secondary (connections
frank-large/frank-primary disabled in openai.api_configs, model table rebuilt,
stale frank-small model_id retargeted to the served alias). Documented the
auth-gated API surface (anonymous probes all 401; keys per account).
Authentik roster gains artem (pk 16) in the human-user template.
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
The local username/password path is retired (ENABLE_LOGIN_FORM +
ENABLE_PASSWORD_AUTH false, ui.enable_login_form DB override false) and
qa-e2e-admin is out of the authentik Admins group. Documented in the SSO
section and History.
The LLM backends section said two ROCmFPX endpoints; the fleet now serves
three (frank-large/medium/small) with different runtimes. Rewrite it, fix the
gotcha that claimed host.containers.internal does not resolve on a bridge
(it does — verified end-to-end for both on-host endpoints), and add a History
entry for the 2026-09-21 model-picker rebuild.
The per-key config DB already pointed web search at SearXNG (engine=searxng,
query URL http://192.168.0.143:8081/search), but only inside the data volume.
The quadlet now carries the same values as seeding env vars, so a fresh data
dir comes up wired, and the doc records the access path: the internal
host-published port, not the search. vhost — the Anubis->Authentik guard
chain is browser-only, and the bridge egress sits in SearXNG's pass_ip, so
the 4-req/IP/hour JSON cap never applies.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
Two gaps behind the same near-miss.
1. **No spec covered hermes.franksdank.cloud** — the one guarded vhost whose
upstream is a host address (`reverse_proxy 192.168.0.143:8787`) rather than a
container name, because its service is host-networked. That is how a
loopback-only bind could take the vhost down with a 502 and leave every gate
green. tests/e2e/specs/17-hermes-vhost.spec.ts asserts what a dead upstream
violates: no 5xx through the guard chain, no Caddy error page, and no app
bytes to an unauthenticated raw client. It does not assert SSO-silence, since
this app has its own password login.
2. **Specs are matched by number prefix, not globbed.** A spec file whose prefix
is in no project silently never runs — which is exactly what happened to the
new one until it was registered. The prefix is registered now, the comment in
playwright.config.ts says why, and tests/e2e_scaffold.bats fails if any
spec file matches no project.
The hermes-webui seam assertion is corrected too: it asserted the loopback bind
I wrongly imposed, and now asserts the real contract — the unit's bind must serve
the address the Caddyfile fronts it on.
ComfyUI's page described a compose service and a systemd unit that shelled out
to `podman compose up`. It now documents the Quadlet, the keep-groups + udev
mechanism (verified inside the running container: torch sees Radeon 8060S), the
9,146-restart incident that forced the move, and the trap that caused it — a
stale container outlives the compose edit that should have replaced it.
The observability page gains the audit result: four jobs read "down" and three
of them always will, because they scrape HTML pages or a /metrics that does not
exist. The page names each error, says plainly that none is an outage, and
refuses the tempting fix (deleting the job makes an absent scrape look like a
healthy one) in favour of the remedy it already recommended: a blackbox exporter
in probe mode, and Gitea's metrics enabled or the job repointed.
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
Raised /swap.img from 8 GiB to 64 GiB on Dankclaw. Host-level state with no
repo file, so it was invisible to the repo until now: fstab line 12 drives the
generated swap.img.swap unit, ext4, fallocated hole-free, swappiness 60,
zswap off. 748 GiB still free on /.
The reusable part is the ordering. The old file held 4 GiB of live pages and
MemAvailable read 73 GiB while only ~5 GiB was genuinely free -- the pinned
model weights are counted as reclaimable page cache, so the headline lies by
the size of the model. Swapping off before the replacement is online would have
pushed those pages into RAM that does not exist. Creating and enabling the new
swap first let them migrate instead: swapoff took 65 s and the host never
wobbled.
New page serverAppConfigs-docs/Dankclaw/Swap Configuration.md, linked from
the index, plus a pointer in the AGENTS.md memory-model section so nobody
reasons about headroom without checking swapon.
Old 8 GiB file staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/
pending approval to purge.
Deployed live at 91e9410 (gluetun+marreta+caddy), authentik provider pk 13,
guard chain verified end-to-end. Documented that Laravel APP_KEY must carry the
base64: prefix — a bare base64 key 500s every app route with 'Unsupported
cipher or incorrect key length' while /up still passes.
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
Post-change doc reconciliation for the hermes vhosts + DNS deletion batch:
- Hermes-Dashboard.md: access is now the hermesadmin.franksdank.cloud vhost
(direct Caddy proxy to 192.168.0.143:9119, no guard chain — exempt because the
dashboard enforces its own basic-auth login), plus the raw LAN port. The old
text claimed an Authentik-guarded public vhost, which was decommissioned.
- Hermes-WebUI.md: description updated — the webui is now also reachable via the
unguarded hermes.franksdank.cloud vhost (its own password auth is the gate).
- LAN-DNS.md: "add a name in two places" corrected to a two-or-three-step
decision; internal-only names (prometheus/loki/grafana/seerr) deliberately get
no public A record and resolve on the LAN only, and DNS-01 issuance is
unaffected because the challenge is a TXT record in the same zone.
- Seerr.md: DNS row corrected — addn-hosts only since 2026-09-02; NXDOMAIN on WAN.
- hermes-dashboard.container: unit comment no longer claims Authentik termination.
No behavior change in this commit; tests 233/233 green (docs-completeness seam included).
New page Architecture/Container Network Namespaces.md: which local services are
addressable from which Podman network namespace on Dankclaw (rootless podman +
netavark). Verified live 2026-09-02: gateway/webui/CDP run in the host netns and
dial loopback; hermes-dashboard is the only Hermes container on a bridge network
(dankclaw_default, 10.89.1.0/24 gw 10.89.1.1) and can dial published ports only —
unpublished host-netns listeners (signal-cli :8080, webui :8787) are refused from
that namespace even though both bind 0.0.0.0.
Corrected while verifying the above:
- Hermes.md: data root is /home/dank/Containers/hermes/data (not the retired
/home/dank/hermes-data); gateway dials the Signal daemon on loopback; the
daemon itself is a host user unit (signal-cli-daemon.service) bound to
0.0.0.0:8080, so the Signal HTTP API is LAN-reachable.
- Hermes-Dashboard.md: it is a Quadlet unit (Network=dankclaw_default), not a
compose service; data root + /opt/data anonymous volume corrected.
- Hermes-WebUI.md: binds 0.0.0.0:8787 in the host netns (verified: unauthenticated
GET /health from another LAN machine returned 200) — the SSH tunnel is an access
path, not a containment boundary; HealthCmd and data-path corrections.
- Network Architecture.md: no hermes vhost exists in either the tracked or the live
Caddyfile (hermes.franksdank.cloud resolves but routes nowhere); SearXNG exits the
tunnel via the shared named network searxng.network, not service:gluetun.
Docs only — deploy is a no-op for this change.
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
Adds a digest-pinned headless-shell Chrome DevTools Protocol daemon as a
boot-persistent quadlet unit (chrome-devtools-protocol.container). The
entrypoint override drops the image's socat front so the CDP endpoint binds
loopback-only on 127.0.0.1:9222 in the host network namespace; only the two
host-network consumers (Signal gateway, webui server) can drive it.
hermes-webui flips Network=dankclaw_default -> Network=host so its agent
process shares the daemon's namespace (0.0.0.0:8787 bind is LAN-facing as
before; no exposure change). deploy_hermes installs the new unit and starts
it if inactive without restarting in-flight browser sessions.
Verified live 2026-08-31: guarded path serves the Anubis JS challenge at
HTTP 200 to non-browser API clients (status-only consumers silently get
HTML); direct published port 8081 from pass_ip sources returns real JSON.
Decision: no second agent-only instance — same engines and same
rate-sensitive VPN exit IP would see double request volume for zero gain.
Frank