caddy + pihole: make both boot-persistent and self-healing
Caddy left compose.yaml for a quadlet unit (caddy.service): joins dankclaw_default for container-name upstreams, Pull=never for the local-only image, Porkbun credentials via /home/dank/Containers/caddy/env written by deploy from .env, bare-IP-catch-all health check. deploy_caddy installs, enables, and starts the unit after retiring the bare container (state is in host volumes). Pi-hole: the repo now owns the sysctl drop-in (dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53); deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted drop-ins in the home trash, and applies 'sysctl --system' (the boot code path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart counter 1645) caused by a second drop-in resetting the value to 80. Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions repointed from the removed compose service to the quadlet; pihole.bats pins the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/ and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack, work log.
This commit is contained in:
Frank
@@ -55,9 +55,9 @@ repo owns their scripts, units, and knowledge base (see
|
||||
|
||||
## Boot Persistence
|
||||
|
||||
The Gitea database and server, the SearXNG trio (gluetun, searxng, valkey), and open-webui run as Quadlet units, so they start at boot. Gitea's web UI is reverse-proxied by the canonical Caddy compose service; the former standalone Gitea Caddy unit is retired. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent.
|
||||
The Gitea database and server, the SearXNG trio (gluetun, searxng, valkey), open-webui, Pi-hole, and Caddy run as Quadlet units, so they start at boot. Caddy left `compose.yaml` on 2026-08-26 after a reboot exposed that bare compose containers never come back; see [[Caddy]] for the unit and the incident. The former standalone Gitea Caddy unit is retired. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent — per AGENTS.md (Boot Persistence & Self-Healing) they each need a quadlet before they may be considered reliably deployed.
|
||||
|
||||
Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw` and open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]); Caddy is deployed through `scripts/deploy.sh dankclaw --service caddy`.
|
||||
Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`, and Pi-hole with `scripts/deploy-pihole-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]); Caddy is deployed through `scripts/deploy.sh dankclaw --service caddy`.
|
||||
|
||||
## Routing Rules
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Caddy
|
||||
|
||||
Caddy is the single reverse proxy for both hosts. It runs as a container on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
|
||||
Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
|
||||
|
||||
**Docs:** https://caddyserver.com/docs
|
||||
|
||||
@@ -8,6 +8,7 @@ Caddy is the single reverse proxy for both hosts. It runs as a container on Dank
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
|
||||
| Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
|
||||
| Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
|
||||
| Data | `/home/dank/Containers/caddy/data` (certs) and `.../config` (ACME state) |
|
||||
@@ -15,7 +16,19 @@ Caddy is the single reverse proxy for both hosts. It runs as a container on Dank
|
||||
| TLS | Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost) |
|
||||
| Runs on | Dankclaw only — Dankserver has no Caddy |
|
||||
|
||||
The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before `up`. See [[Deployment Flow]].
|
||||
The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].
|
||||
|
||||
## Quadlet (boot persistence)
|
||||
|
||||
Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").
|
||||
|
||||
- **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
|
||||
- **Network:** `dankclaw_default` (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
|
||||
- **Published ports:** 80/tcp, 443/tcp
|
||||
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
|
||||
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
|
||||
- **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
|
||||
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs, enables, and starts the unit
|
||||
|
||||
## The porkbun image
|
||||
|
||||
@@ -34,7 +47,7 @@ acme_dns porkbun {
|
||||
}
|
||||
```
|
||||
|
||||
compose passes both variables from the host `.env`. Keep the real keys in `.env` only; `.env.example` holds placeholders. The Porkbun account holds only `franksdank.cloud`.
|
||||
The deploy writes both variables from the host `.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `.env.example` (and `dankclaw/caddy/env.example`) hold placeholders. The Porkbun account holds only `franksdank.cloud`.
|
||||
|
||||
## Vhost map
|
||||
|
||||
@@ -83,6 +96,7 @@ Certificates and ACME state live in the caddy data volume. Back it up with the s
|
||||
|
||||
## Gotchas
|
||||
|
||||
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to `compose.yaml`.
|
||||
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
|
||||
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
|
||||
- A new vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
|
||||
|
||||
@@ -9,6 +9,7 @@ Pi-hole is the LAN DNS server for the homelab. It runs as a quadlet container on
|
||||
| `pihole` quadlet | `dankclaw/pihole/quadlet/pihole.container` | Boot-persistent container unit |
|
||||
| FTL | `docker.io/pihole/pihole:latest` | DNS engine (a dnsmasq fork) + ad blocking |
|
||||
| `addn-hosts` | `dankclaw/pihole/addn-hosts` | One A record per Caddy vhost |
|
||||
| sysctl drop-in | `dankclaw/pihole/sysctl/99-dnsmasq.conf` (repo) → `/etc/sysctl.d/` | Rootless port-53 threshold, applied at boot |
|
||||
| env file | `/home/dank/Containers/pihole/env` (host) | `ServerIP`, `TZ`, `DNS1`, `DNS2`, `WEBPASSWORD` |
|
||||
| Web UI | `pihole.franksdank.cloud` (Caddy → 8083) | Dashboard + admin, family login |
|
||||
|
||||
@@ -49,7 +50,7 @@ The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `D
|
||||
|
||||
## Host prerequisites
|
||||
|
||||
1. `sysctl net.ipv4.ip_unprivileged_port_start=53` (persisted in `/etc/sysctl.d/99-dnsmasq.conf`) — rootless Podman may then publish port 53.
|
||||
1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any later-sorted drop-in that resets the value in the home trash, and applies it with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** a second drop-in (`99-rootless-ports.conf`, sorted after) reset the value to 80. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until the conflict was removed.
|
||||
2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns.
|
||||
|
||||
Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
|
||||
@@ -62,6 +63,7 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
|
||||
| A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] |
|
||||
| The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit |
|
||||
| Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
|
||||
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: exactly one file under `/etc/sysctl.d/` may set `ip_unprivileged_port_start`, and it must say 53. Redeploy pihole — `deploy_pihole` reinstalls the drop-in and neutralizes conflicts |
|
||||
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
|
||||
|
||||
## See Also
|
||||
|
||||
@@ -30,3 +30,10 @@ timestamp: 2026-09-14
|
||||
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
|
||||
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
|
||||
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
|
||||
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): a later-sorted sysctl drop-in (`99-rootless-ports.conf`) reset `net.ipv4.ip_unprivileged_port_start` to 80, so the rootless port-53 publish failed with `bind: permission denied`.
|
||||
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
|
||||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
|
||||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting later-sorted drop-ins in the home trash, and applies `sysctl --system` (the boot code path).
|
||||
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/enables/starts the quadlet (retiring the bare container; all state is in host volumes); `deploy_pihole` enforces the sysctl prerequisite.
|
||||
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
|
||||
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.
|
||||
|
||||
Reference in New Issue
Block a user