Files
serverAppConfigs/docs/Infrastructure/Caddy.md
T
Frank 8660209c35 caddy + pihole: make both boot-persistent and self-healing
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).

Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.

Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
2026-08-26 19:52:57 -07:00

7.7 KiB

Caddy

Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (caddy.service) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.

Docs: https://caddyserver.com/docs

Facts (live inventory)

Item Value
Runtime Quadlet unit caddy.service (dankclaw/caddy/quadlet/caddy.container) — boot-persistent, self-healing
Image franksdank/caddy-porkbun:latest (local build, dankclaw/caddy/Containerfile)
Config /home/dank/Containers/caddy/Caddyfile (bind mount, rendered by deploy)
Data /home/dank/Containers/caddy/data (certs) and .../config (ACME state)
Ports 80, 443 on 192.168.0.143
TLS Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost)
Runs on Dankclaw only — Dankserver has no Caddy

The Caddyfile in git (dankclaw/Caddyfile) carries a REPLACE_WITH_HASH placeholder. scripts/deploy.sh renders the real basicauth hash from the host .env (CADDY_BASICAUTH_HASH) into the mounted path before start. See Deployment Flow.

Quadlet (boot persistence)

Caddy left compose.yaml on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").

  • Install path on Dankclaw: ~/.config/containers/systemd/caddy.container (unit caddy.service)
  • Network: dankclaw_default (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
  • Published ports: 80/tcp, 443/tcp
  • Env file: /home/dank/Containers/caddy/env (PORKBUN_API_KEY, PORKBUN_API_SECRET_KEY) — the deploy writes it from the checkout's .env; secrets never enter the repo (placeholders in dankclaw/caddy/env.example)
  • Restart: Restart=always, RestartSec=5; the health check probes the bare-IP catch-all (302) inside the container
  • Image policy: Pull=never — the image is built on-host and exists nowhere else; a boot must not stall on a registry
  • Deploy: ./scripts/deploy.sh dankclaw --service caddy — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs, enables, and starts the unit

The porkbun image

Stock Caddy cannot solve the ACME challenge because the router exposes no inbound port 80/443. The local image adds the github.com/caddy-dns/porkbun module, which answers the DNS-01 challenge by creating and deleting a _acme-challenge TXT record through the Porkbun API. No inbound port is needed. Rebuild only when Caddy or the module needs an update:

podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .

The Caddyfile references the credentials through the environment:

acme_dns porkbun {
    api_key {env.PORKBUN_API_KEY}
    api_secret_key {env.PORKBUN_API_SECRET_KEY}
}

The deploy writes both variables from the host .env into /home/dank/Containers/caddy/env, which the quadlet reads via EnvironmentFile. Keep the real keys in .env only; .env.example (and dankclaw/caddy/env.example) hold placeholders. The Porkbun account holds only franksdank.cloud.

Vhost map

Every vhost uses the flat franksdank.cloud convention. The full list lives in dankclaw/Caddyfile. Routing by host:

Upstream Reach
Dankclaw services (compose network) by container name, e.g. comfyui:8188
Dankclaw quadlet services by published LAN port, e.g. 192.168.0.143:3000 (Gitea)
Dankserver services by published LAN port, e.g. 192.168.0.235:8085 (RomM)

The bare-IP catch-all :80 302-redirects to https://dankclaw.franksdank.cloud/. The legacy name dankserver.franksdank.cloud 302-redirects there too. plex.franksdank.casa is the only plain-HTTP vhost; it cannot get a DNS-01 cert because franksdank.casa is not in the Porkbun account. The canonical Plex name is plex.franksdank.cloud. Both Plex vhosts dial PMS over TLS upstream (transport http { tls_insecure_skip_verify } — in this Caddy build the flag is a bare token inside the transport block, not a tls {} sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). insecure_skip_verify applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at /web/; / returns the XML MediaContainer index.

Basic auth

The Caddyfile in git carries a bare placeholder; deploy.sh substitutes the full hash from the host .env (CADDY_BASICAUTH_HASH):

chat.franksdank.cloud {
    basicauth {
        dank REPLACE_WITH_HASH
    }
    reverse_proxy 192.168.0.143:3005
}

After rendering, the line reads dank $2a$14$xxxx… (one complete hash). Do not put a $2a$14$ prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). tests/deployment_flow.bats pins the render contract.

The homepage (and SearXNG, Hermes, Prometheus, RomM, Copyparty) use forward_auth (Authentik SSO) rather than basicauth; see Homepage and Authentik. The basicauth hash above is still rendered for the vhosts that keep it (Grafana, Pi-hole).

Operations

Validate before a reload:

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile

The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile change hot-reloads within seconds via the file watcher. deploy.sh also issues an explicit reload after rendering. Manual reload from the host:

podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile

Certificates and ACME state live in the caddy data volume. Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01.

Gotchas

  • A bare compose container is not boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because restart: unless-stopped does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to compose.yaml.
  • admin off disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). tests/caddyfile.bats forbids admin off.
  • The basicauth template must stay a bare REPLACE_WITH_HASH token (no $2a$14$ prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
  • A new vhost needs three things: a Pi-hole addn-hosts line, a Porkbun A record, and the Caddyfile block. See LAN-DNS.
  • The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in LAN-DNS.
  • Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.

See Also