serverAppConfigs/serverAppConfigs-docs/Infrastructure/Caddy.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Caddy is the single reverse proxy for both hosts.	infrastructure caddy	2026-08-29

Caddy

Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (caddy.service) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and gates the guarded vhosts through the two-tier chain — Anubis bot-challenge first, then Authentik — rather than through HTTP basic auth (dankclaw/Caddyfile:14-24). Basic auth survives on exactly one vhost today: the Pi-hole admin UI (dankclaw/Caddyfile:166-171).

Docs: https://caddyserver.com/docs

Facts (live inventory)

Item	Value
Runtime	Quadlet unit caddy.service (dankclaw/caddy/quadlet/caddy.container) — boot-persistent, self-healing
Image	franksdank/caddy-porkbun:latest (local build, dankclaw/caddy/Containerfile)
Config	/home/dank/Containers/caddy/Caddyfile (bind mount, rendered by deploy)
Data	/home/dank/Containers/caddy/data — Caddy's whole store: data/caddy/certificates/ (leaf certs) and data/caddy/acme/ (the ACME account). .../config holds only config/caddy/autosave.json; it is not where ACME state lives (host ls, 2026-10-18)
Ports	80, 443 on 192.168.0.143
TLS	Let's Encrypt, DNS-01 via the Porkbun module — one certificate per name, not a wildcard: 31 per-name directories under /home/dank/Containers/caddy/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/ (host ls, 2026-10-18)
Runs on	Dankclaw only — Dankserver has no Caddy

The Caddyfile in git (dankclaw/Caddyfile) carries a REPLACE_WITH_HASH placeholder. scripts/deploy.sh renders the real basicauth hash from the host .env (CADDY_BASICAUTH_HASH) into the mounted path before start. See Deployment Flow.

Quadlet (boot persistence)

Caddy left compose.yaml on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").

• Install path on Dankclaw: ~/.config/containers/systemd/caddy.container (unit caddy.service)
• Network: dankclaw_default — the long-lived bridge (10.89.1.0/24, gateway 10.89.1.1, podman network inspect dankclaw_default, 2026-10-18) that the old compose stack created; the Caddyfile's container-name upstreams must keep resolving on it. dankclaw/compose.yaml no longer exists in the repo or on the host (ls, 2026-10-18), so from now on the quadlet units that name this network are the only thing that creates it at boot.
• Published ports: 80/tcp, 443/tcp
• Env file: /home/dank/Containers/caddy/env (PORKBUN_API_KEY, PORKBUN_API_SECRET_KEY) — the deploy creates it from the checkout's dankclaw/.env if it is missing and then fails if either variable is absent (scripts/deploy-remote.sh:405-412); secrets never enter the repo (placeholders in dankclaw/caddy/env.example)
• Restart: Restart=always, RestartSec=5; the health check probes the bare-IP catch-all inside the container, which answers 308 (not 302) — curl -sI http://127.0.0.1:80/ → 308, Location: https://www.franksdank.cloud/ (host, 2026-10-18; dankclaw/Caddyfile:60-62)
• Image policy: Pull=never — the image is built on-host and exists nowhere else; a boot must not stall on a registry
• Enable state: systemctl --user is-enabled caddy reports generated — that is the healthy state for a quadlet. The podman generator creates the default.target.wants symlink from the unit's [Install] section at every user-manager start (linger covers boot); an explicit systemctl enable fails on generated units, so the deploy does not run it
• Deploy: ./scripts/deploy.sh dankclaw --service caddy → deploy_caddy() (scripts/deploy-remote.sh:383-429) renders the Caddyfile, rebuilds the image when the Containerfile hash changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), installs the quadlet, restarts the unit, and issues podman exec caddy caddy reload afterwards

The porkbun image

Stock Caddy cannot solve the ACME challenge because the router exposes no inbound port 80/443. The local image adds the github.com/caddy-dns/porkbun module, which answers the DNS-01 challenge by creating and deleting a _acme-challenge TXT record through the Porkbun API. No inbound port is needed. Rebuild only when Caddy or the module needs an update:

podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .

The Caddyfile references the credentials through the environment:

acme_dns porkbun {
    api_key {env.PORKBUN_API_KEY}
    api_secret_key {env.PORKBUN_API_SECRET_KEY}
}

The deploy writes both variables from dankclaw/.env into /home/dank/Containers/caddy/env, which the quadlet reads via EnvironmentFile. Keep the real keys in .env only; dankclaw/.env.example and dankclaw/caddy/env.example hold placeholders (there is no repository-root env.example — find, 2026-10-18). The account holds franksdank.cloud and not franksdank.casa — that asymmetry is why plex.franksdank.casa stays plain HTTP (dankclaw/Caddyfile:204-207); whether it holds any other domain is not checkable from this repo (unverified 2026-10-18).

Vhost map

Every vhost uses the flat franksdank.cloud convention. The full list lives in dankclaw/Caddyfile. Routing by host:

Upstream	Reach
Dankclaw services on dankclaw_default	by container name: authentik-server:9000, anubis:8923, grafana:3000, prometheus:9090, loki:3100, seerr:5055 (dankclaw/Caddyfile:94,75,376,381,366,371)
Dankclaw quadlet services on isolated networks	by published LAN port: 192.168.0.143:3000 (Gitea), :3005 (OpenWebUI), :8081 (SearXNG via Gluetun), :8082 (Marreta), :8083 (Pi-hole), :9119 (Hermes dashboard)
Host-network containers on Dankclaw	host.containers.internal:8787 (Hermes WebUI) — a host-netns bind is not a published port, so netavark's host address is the only upstream that reaches it (dankclaw/Caddyfile:179-188)
Dankserver services	by published LAN port, e.g. 192.168.0.235:8085 (RomM), :32400 (Plex), :3923 (Copyparty)
FrankBook-Pro services	by published LAN port on 192.168.0.112 (the *arr stack and qBittorrent; those ports publish on that host's Gluetun netns owner)

ComfyUI is not in this table: it has no vhost. It publishes 0.0.0.0:8188 and is reached at comfyui.franksdank.cloud:8188 through the Pi-hole A record, with no Caddy and no authentication on the LAN (dankclaw/comfyui/quadlet/comfyui.container:24-30).

The bare-IP catch-all :80 308-redirects to https://www.franksdank.cloud/ (dankclaw/Caddyfile:60-62, status measured 2026-10-18), and so do the two legacy names dankclaw.franksdank.cloud and dankserver.franksdank.cloud (dankclaw/Caddyfile:81-88) — dankclaw. is a redirect target of the past, not of :80. plex.franksdank.casa is the only plain-HTTP vhost; it cannot get a DNS-01 cert because franksdank.casa is not in the Porkbun account. The canonical Plex name is plex.franksdank.cloud. Both Plex vhosts dial PMS over TLS upstream (transport http { tls_insecure_skip_verify } — in this Caddy build the flag is a bare token inside the transport block, not a tls {} sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). The version line holds: the container on Dankserver is lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325 (podman ps, 2026-10-18). insecure_skip_verify applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at /web/; / returns the XML MediaContainer index.

Basic auth

The Caddyfile in git carries a bare placeholder; deploy.sh substitutes the full hash from the host .env (CADDY_BASICAUTH_HASH):

pihole.franksdank.cloud {
    basicauth {
        dankuser REPLACE_WITH_HASH
    }
    reverse_proxy 192.168.0.143:8083
}

After rendering, the line reads dankuser <hash> (one complete hash) — tests/deployment_flow.bats:96-99 pins the dankuser name and the single-hash form, and tests/caddyfile.bats:75 pins the bare dankuser REPLACE_WITH_HASH template. Do not put a $2a$14$ prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; tests/caddyfile.bats:67-72 forbids the prefix). tests/deployment_flow.bats pins the render contract.

Everything else authenticates, not basicauth. The guarded set — www. (homepage), search., grafana., prometheus., loki., seerr., marreta., romm., files. — proxies all traffic to anubis:8923, which verifies the client and forwards it back to Caddy's internal listener http://:8016; Authentik's proxy-provider mode serves the login inline (this build exposes no forward_auth endpoint) and forwards authenticated traffic to http://:8017, where the request is routed straight to its app (dankclaw/Caddyfile:14-24,283-423). The exempt-by-construction set never enters that chain: auth. (the IdP itself), ai. and gitea. (own logins), hermes. and hermesadmin. (app-enforced passwords), pihole. (the basicauth edge above), and the media/*arr vhosts. Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik (dankclaw/Caddyfile:122-132). See Homepage and Authentik.

Operations

Validate before a reload:

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile

The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile change hot-reloads within seconds via the file watcher. deploy.sh also issues an explicit reload after rendering. Manual reload from the host:

podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile

Certificates and ACME state live in the caddy data volume (data/caddy/). Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01. The store also still holds leaf certificates for names the Caddyfile no longer defines — sonarr., chat., comfyui., ntfy., jellyfin. (.franksdank.cloud) each have a directory there (host ls, 2026-10-18) — retired vhosts leave certificates behind, which is harmless but is the record that those names once existed.

Gotchas

• A bare compose container is not boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because restart: unless-stopped does not survive shutdown's explicit stop. It now runs only from its quadlet unit, and the repo has no dankclaw/compose.yaml left to re-add it to (ls, 2026-10-18).
• admin off disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). tests/caddyfile.bats forbids admin off.
• The basicauth template must stay a bare REPLACE_WITH_HASH token (no $2a$14$ prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
• A public vhost needs three things: a Pi-hole addn-hosts line, a Porkbun A record, and the Caddyfile block. See LAN-DNS. The deliberate exception is an internal-only name: hermes.franksdank.cloud and hermesadmin.franksdank.cloud are in addn-hosts and the Caddyfile but have no public record, which is what keeps them LAN-only — DNS-01 issuance needs no public A record (dankclaw/Caddyfile:173-179).
• The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in LAN-DNS. (unverified 2026-10-18: hairpin behaviour is only observable from a WAN-side client; nothing on the LAN or in this repo can prove it)
• Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
• transport http {versions h2c 1.1} on a vhost makes Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it and Caddy turns the failure into a 502 while the same port answers 200 over plain HTTP/1.1. Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1 (dankclaw/Caddyfile:248-251 carries no transport block), Plex PMS = TLS (dankclaw/Caddyfile:221-227; PMS drops plain HTTP off-loopback — the tls_insecure_skip_verify hop is measured in tests/caddyfile.bats, and no vhost in the current Caddyfile sets versions h2c).

See Also

• LAN-DNS — how names resolve on the LAN and WAN
• Network Architecture — topology, access policy
• Homepage
• Dankclaw Stack