pihole: also neutralize the /etc/sysctl.conf sysctl override

systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the
stale ip_unprivileged_port_start=80 line there beat every drop-in and kept
the runtime value at 80 after the first deploy pass. deploy_pihole now
comments out an active line in /etc/sysctl.conf (idempotent) and generalizes
the sysctl.d conflict scan to any file setting a value above 53.
This commit is contained in:
Frank
2026-08-26 20:08:30 -07:00
parent 8660209c35
commit f3c1ac1823
4 changed files with 30 additions and 12 deletions
+2 -2
View File
@@ -50,7 +50,7 @@ The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `D
## Host prerequisites
1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any later-sorted drop-in that resets the value in the home trash, and applies it with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** a second drop-in (`99-rootless-ports.conf`, sorted after) reset the value to 80. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until the conflict was removed.
1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any drop-in that sets a value above 53 in the home trash, comments out a stale active line in `/etc/sysctl.conf`, and applies everything with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** the value was reset to 80 by *two* sources — a later-sorted drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl processes **after** all of `sysctl.d/` and therefore overrides every drop-in. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until both conflicts were removed.
2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns.
Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
@@ -63,7 +63,7 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
| A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] |
| The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit |
| Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: exactly one file under `/etc/sysctl.d/` may set `ip_unprivileged_port_start`, and it must say 53. Redeploy pihole — `deploy_pihole` reinstalls the drop-in and neutralizes conflicts |
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
## See Also
+2 -2
View File
@@ -30,10 +30,10 @@ timestamp: 2026-09-14
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): a later-sorted sysctl drop-in (`99-rootless-ports.conf`) reset `net.ipv4.ip_unprivileged_port_start` to 80, so the rootless port-53 publish failed with `bind: permission denied`.
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting later-sorted drop-ins in the home trash, and applies `sysctl --system` (the boot code path).
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/enables/starts the quadlet (retiring the bare container; all state is in host volumes); `deploy_pihole` enforces the sysctl prerequisite.
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.