---
type: reference
description: "Caddy is the single reverse proxy for both hosts."
tags:
  - infrastructure
  - caddy
timestamp: 2026-11-03
---

# Caddy

Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and gates the guarded vhosts through the two-tier chain — Anubis bot-challenge first, then Authentik — rather than through HTTP basic auth (`dankclaw/Caddyfile:14-24`). Basic auth survives on exactly one vhost today: the Pi-hole admin UI (`dankclaw/Caddyfile:166-171`).

**Docs:** https://caddyserver.com/docs

## Facts (live inventory)

| Item | Value |
|------|-------|
| Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
| Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
| Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
| Data | `/home/dank/Containers/caddy/data` — Caddy's whole store: `data/caddy/certificates/` (leaf certs) and `data/caddy/acme/` (the ACME account). `.../config` holds only `config/caddy/autosave.json`; it is not where ACME state lives (host `ls`, 2026-10-18) |
| Ports | 80, 443 on 192.168.0.143 |
| TLS | Let's Encrypt, DNS-01 via the Porkbun module — one certificate per name, not a wildcard: 31 per-name directories under `/home/dank/Containers/caddy/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/` (host `ls`, 2026-10-18) |
| Runs on | Dankclaw only — Dankserver has no Caddy |

The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].

## Quadlet (boot persistence)

Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").

- **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
- **Network:** `dankclaw_default` — the long-lived bridge (`10.89.1.0/24`, gateway `10.89.1.1`, `podman network inspect dankclaw_default`, 2026-10-18) that the old compose stack created; the Caddyfile's container-name upstreams must keep resolving on it. `dankclaw/compose.yaml` no longer exists in the repo or on the host (`ls`, 2026-10-18), so from now on the quadlet units that name this network are the only thing that creates it at boot.
- **Published ports:** 80/tcp, 443/tcp
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy creates it from the checkout's `dankclaw/.env` if it is missing and then fails if either variable is absent (`scripts/deploy-remote.sh:405-412`); secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all inside the container, which answers **308** (not 302) — `curl -sI http://127.0.0.1:80/` → `308`, `Location: https://www.franksdank.cloud/` (host, 2026-10-18; `dankclaw/Caddyfile:60-62`)
- **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
- **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` → `deploy_caddy()` (`scripts/deploy-remote.sh:383-429`) renders the Caddyfile, rebuilds the image when the Containerfile hash changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), installs the quadlet, restarts the unit, and issues `podman exec caddy caddy reload` afterwards

## The porkbun image

Stock Caddy cannot solve the ACME challenge because the router exposes no inbound port 80/443. The local image adds the `github.com/caddy-dns/porkbun` module, which answers the DNS-01 challenge by creating and deleting a `_acme-challenge` TXT record through the Porkbun API. No inbound port is needed. Rebuild only when Caddy or the module needs an update:

```
podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .
```

The Caddyfile references the credentials through the environment:

```
acme_dns porkbun {
    api_key {env.PORKBUN_API_KEY}
    api_secret_key {env.PORKBUN_API_SECRET_KEY}
}
```

The deploy writes both variables from `dankclaw/.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `dankclaw/.env.example` and `dankclaw/caddy/env.example` hold placeholders (there is no repository-root `env.example` — `find`, 2026-10-18). The account holds `franksdank.cloud` and not `franksdank.casa` — that asymmetry is why `plex.franksdank.casa` stays plain HTTP (`dankclaw/Caddyfile:204-207`); whether it holds any other domain is not checkable from this repo *(unverified 2026-10-18)*.

## Vhost map

Every vhost uses the flat `franksdank.cloud` convention. The full list lives in `dankclaw/Caddyfile`. Routing by host:

| Upstream | Reach |
|----------|-------|
| Dankclaw services on `dankclaw_default` | by container name: `authentik-server:9000`, `anubis:8923`, `grafana:3000`, `prometheus:9090`, `loki:3100`, `seerr:5055` (`dankclaw/Caddyfile:94,75,376,381,366,371`) |
| Dankclaw quadlet services on isolated networks | by published LAN port: `192.168.0.143:3000` (Gitea), `:3005` (OpenWebUI), `:8081` (SearXNG via Gluetun), `:8082` (Marreta), `:8083` (Pi-hole), `:9119` (Hermes dashboard) |
| Host-network containers on Dankclaw | `host.containers.internal:8787` (Hermes WebUI) — a host-netns bind is not a published port, so netavark's host address is the only upstream that reaches it (`dankclaw/Caddyfile:179-188`) |
| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM), `:32400` (Plex), `:3923` (Copyparty) |
| FrankBook-Pro services | by published LAN port on `192.168.0.112` (the *arr stack and qBittorrent; those ports publish on that host's Gluetun netns owner) |

ComfyUI is **not** in this table: it has no vhost. It publishes `0.0.0.0:8188` and is reached at `comfyui.franksdank.cloud:8188` through the Pi-hole A record, with no Caddy and no authentication on the LAN (`dankclaw/comfyui/quadlet/comfyui.container:24-30`).

The bare-IP catch-all `:80` 308-redirects to `https://www.franksdank.cloud/` (`dankclaw/Caddyfile:60-62`, status measured 2026-10-18), and so do the two legacy names `dankclaw.franksdank.cloud` and `dankserver.franksdank.cloud` (`dankclaw/Caddyfile:81-88`) — `dankclaw.` is a redirect target of the past, not of `:80`. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). The version line holds: the container on Dankserver is `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (`podman ps`, 2026-10-18). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.

## Basic auth

The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):

```
pihole.franksdank.cloud {
    basicauth {
        dankuser REPLACE_WITH_HASH
    }
    reverse_proxy 192.168.0.143:8083
}
```

After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.

Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`, `notebook.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile`: the header comment and the two internal listeners). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), the media/*arr vhosts, and `audiobooks.` — Audiobookshelf enforces its own auth (local accounts plus authentik OIDC) and must answer native app clients, podcast RSS and OPDS, which the chain structurally cannot deliver: it answers a client that solves no JavaScript proof-of-work with challenge HTML wherever that client expects JSON or a socket payload ([[Audiobookshelf]]). Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik. See [[Homepage]] and [[Authentik]].

## Guard a new vhost

Order is the whole procedure: the IdP row has to exist **before** the route flips, or
authenticated traffic reaches the outpost with no matching provider and the user gets
authentik's error page instead of the app.

1. **Provider first.** A proxy provider plus an application, claimed by the embedded
   outpost. No blueprint owns this state — it lives only in the IdP's database, which is
   why it is easy to forget:

   ```
   ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
   from authentik.providers.proxy.models import ProxyProvider
   from authentik.core.models import Application
   from authentik.outposts.models import Outpost
   NAME, EXT = "example", "https://example.franksdank.cloud/"
   pp = ProxyProvider.objects.filter(name=f"{NAME}-proxy").first()
   if pp is None:
       pp = ProxyProvider.objects.create(name=f"{NAME}-proxy", external_host=EXT,
                                         internal_host="http://caddy:8017")
   if not Application.objects.filter(provider=pp).exists():
       Application.objects.create(name=f"{NAME.title()} Guard", slug=f"{NAME}-guard",
                                  provider=pp)
   o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
   o.providers.add(pp)   # an unclaimed provider is invisible to the outpost
   print(pp.name, pp.external_host, o.providers.count())
   PY
   ```

   `external_host` is the exact vhost URL **with the trailing slash**: the embedded
   outpost routes on the ORIGINAL `Host` header alone, so anything else matches zero
   providers. Never print a provider's fields wholesale — a proxy provider carries a
   client secret, and it lands in whatever transcript the command runs in.

   `ProxyProvider.objects.create()` is not the API: the serializer's
   `set_oauth_defaults()` never runs, so the row lands with `redirect_uris` and
   `grant_types` empty, and the guard fails *before* any credential — authentik's
   "Redirect URI Error" page, then a bodiless HTTP 400 once the URIs are filled
   by hand. Run `scripts/authentik-proxy-provider-defaults.sh` after the snippet
   (idempotent, covers every provider; `--check` is the census); law and failure
   signatures in [[Authentik Outpost Reference]].
2. **Then the route.** The public site block proxies the whole request to `anubis:8923`
   with `header_up X-Real-Ip {remote_host}`, and the post-auth `http://:8017` listener
   gains a `@<name> host <vhost>` handler pointing at the app — copy `romm`/`files`,
   which are the same shape including a Dankserver LAN-port upstream.
   An app that derives its own origin from the request needs the scheme pinned:
   Audiobookshelf checks every OIDC callback against `x-forwarded-proto://Host`; this
   listener is fed over plain http so Caddy's own `X-Forwarded-Proto` is `http`, and the
   app answers a 400 "Invalid callback URL - must be same-origin" for its own https
   callback (measured 2026-11-03, while this vhost was still guarded — it is exempt
   now, and the same pin sits on its edge block instead).
   The edge for a guarded name is TLS-only, so `header_up X-Forwarded-Proto
   https` is the truthful value ([[Audiobookshelf]]).
3. **Then the name and the audit.** A `dankclaw/pihole/addn-hosts` line (the `pihole`
   seam fails the build if a Caddyfile vhost has no record), a Porkbun A record if the
   name must resolve off-LAN ([[LAN-DNS]]), and the subdomain added to `GUARDED` in
   `tests/e2e/specs/14-guard.spec.ts` with an app-specific leak marker — that spec is
   the deploy gate's proof that the guard, not the app, answers an anonymous client.

## Unguard a vhost

The same three artifacts, in the opposite order. Getting it wrong takes the app
down for browsers while looking healthy:

1. **Route first.** Move the public site block off `anubis:8923` to a direct
   `reverse_proxy` (keeping any origin-pinning `header_up` the app needs), delete
   the `@<name>` handler from the `http://:8017` listener, and deploy. Traffic
   stops entering the auth listener at `:8016`, so the IdP row goes unused — and
   nothing breaks.
2. **Then the IdP rows**, once the route is verified:

   ```
   ssh dankclaw 'podman exec -i authentik-server ak shell' <<'PY'
   from authentik.providers.proxy.models import ProxyProvider
   from authentik.outposts.models import Outpost
   pp = ProxyProvider.objects.filter(name="audiobooks-proxy").first()
   if pp:
       o = Outpost.objects.get(managed="goauthentik.io/outposts/embedded")
       o.providers.remove(pp)
       app = pp.application          # the "<Name> Guard" application row
       if app:
           app.delete()
       pp.delete()
   print("providers now claimed:", Outpost.objects.get(
       managed="goauthentik.io/outposts/embedded").providers.count())
   PY
   ```

   Deleting the provider **before** the route flips is the wrong order: the
   outpost then matches zero providers for a Host that still arrives, and every
   browser gets authentik's error page instead of the app.
3. **Then the audit lists.** Drop the subdomain from `GUARDED` in
   `tests/e2e/specs/14-guard.spec.ts` (and its leak marker), and replace it
   there with an exemption assertion — the deploy gate must keep proving the
   exemption is deliberate, the way `plex` does: app bytes to an anonymous
   client, zero challenge HTML, and the app's own `401` on an unauthenticated
   API call rather than the gate's `200`.

What the exemption costs, said plainly: the app's login endpoint becomes
internet-visible with no rate limiter in front of it unless the edge carries one
(this Caddy build has no `rate_limit` module) or the app ships a lockout (ABS
2.37.2 does not). That is the same exposure `jelly.`, `plex.` and the *arr
vhosts already live with; the mitigation is the credentials.

## Operations

Validate before a reload:

```
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
```

The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile change hot-reloads within seconds via the file watcher. deploy.sh also issues an explicit reload after rendering. Manual reload from the host:

```
podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
```

Certificates and ACME state live in the caddy data volume (`data/caddy/`). Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01. The store also still holds leaf certificates for names the Caddyfile no longer defines — `sonarr.`, `chat.`, `comfyui.`, `ntfy.`, `jellyfin.` (`.franksdank.cloud`) each have a directory there (host `ls`, 2026-10-18) — retired vhosts leave certificates behind, which is harmless but is the record that those names once existed.

## Gotchas

- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit, and the repo has no `dankclaw/compose.yaml` left to re-add it to (`ls`, 2026-10-18).
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
- A public vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]]. The deliberate exception is an **internal-only** name: `hermes.franksdank.cloud` and `hermesadmin.franksdank.cloud` are in `addn-hosts` and the Caddyfile but have no public record, which is what keeps them LAN-only — DNS-01 issuance needs no public A record (`dankclaw/Caddyfile:173-179`).
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]]. *(unverified 2026-10-18: hairpin behaviour is only observable from a WAN-side client; nothing on the LAN or in this repo can prove it)*
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
- `transport http {versions h2c 1.1}` on a vhost makes Caddy send an h2c
  upgrade preface; .NET/Kestrel backends (Jellyfin) reject it and Caddy turns
  the failure into a 502 while the same port answers 200 over plain HTTP/1.1.
  Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1
  (`dankclaw/Caddyfile:248-251` carries no transport block), Plex PMS = TLS
  (`dankclaw/Caddyfile:221-227`; PMS drops plain HTTP off-loopback — the
  `tls_insecure_skip_verify` hop is measured in `tests/caddyfile.bats`, and no
  vhost in the current Caddyfile sets `versions h2c`).

## See Also

- [[LAN-DNS]] — how names resolve on the LAN and WAN
- [[Network Architecture]] — topology, access policy
- [[Homepage]]
- [[Dankclaw Stack]]