Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.
The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).
Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.
Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
Caddy v2.11.4's caddyfile adapter rejects 'tls { insecure_skip_verify }'
inside transport http (unrecognized subdirective '{') and also rejects
the handler-level form. The working form, verified live against PMS with
a throwaway caddy (200 OK), is the bare token:
reverse_proxy 192.168.0.235:32400 {
transport http {
tls_insecure_skip_verify
}
}
The other vhosts (gitea, grafana, search, ...) now use username 'dankuser'
with CADDY_BASICAUTH_HASH, which is set to the same hash as the homepage
(dankpassword.123). So the whole protected surface logs in as
dankuser / dankpassword.123; the operator /admin/ page still uses frank.
Updated the two tests that asserted the old 'dank' username.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
(the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
Per user direction, Caddy runs only on dankclaw. It fronts every
service on both hosts: local services by container name, quadlet
services by published LAN port, dankserver services by published LAN
port (192.168.0.235:PORT).
- dankclaw/Caddyfile: merged vhost list (was: two Caddyfiles, one per
host). gitea's single vhost moves in from the retired gitea-caddy
quadlet (option B). Upstreams fixed to live reality: gitea :3000,
search :8081 (gluetun), chat :3005 (open-webui quadlet), media
services via 192.168.0.235. Retired: llms (open-webui on dankserver
is removed; chat is its successor), the dankserver homepage (302 to
the shared one).
- dankserver/Caddyfile, dankclaw/gitea/Caddyfile: deleted (git rm).
- dnsmasq addn-hosts: every name -> 192.168.0.143 (the one Caddy
host); no .235 records, no llms.
- compose.yaml: removed the openwebui/postgres/gitea services (they
are quadlets now); caddy service mounts the rendered Caddyfile from
/home/dank/Containers/caddy, healthcheck probes the bare-IP
catch-all (302).
- deploy.sh: per-target logic (dankserver = pull-only, health via the
shared Caddy at 192.168.0.143); renders CADDY_BASICAUTH_HASH from
the host .env into the mounted Caddyfile.
- .env.example: CADDY_BASICAUTH_HASH on dankclaw; Porkbun block
dropped from dankserver (its Caddy is gone); SEARXNG_BASE_URL to
the flat name; stale OPENAI_* block dropped (quadlet is
self-contained).
- tests: caddyfile.bats now asserts the single-Caddyfile invariant;
dnsmasq.bats cross-checks vhosts -> .143 and the retired names.
Phase 1 of the local-DNS-name work. Users type a name, not a port.
- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
renamed to flat <service>.franksdank.cloud. Explicit http:// site
addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
from the repo): added the live-only llms.franksdank.cloud vhost
(OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
(the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
the right host IPs); caddyfile.bats naming-convention tests updated.
LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
Frank