From f3c1ac18237d8dff0bc003342fca8a33cf601728 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 26 Aug 2026 20:08:30 -0700 Subject: [PATCH] pihole: also neutralize the /etc/sysctl.conf sysctl override systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the stale ip_unprivileged_port_start=80 line there beat every drop-in and kept the runtime value at 80 after the first deploy pass. deploy_pihole now comments out an active line in /etc/sysctl.conf (idempotent) and generalizes the sysctl.d conflict scan to any file setting a value above 53. --- docs/Infrastructure/Pi-hole.md | 4 ++-- docs/log.md | 4 ++-- scripts/deploy-remote.sh | 28 ++++++++++++++++++++-------- tests/pihole.bats | 6 ++++++ 4 files changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/Infrastructure/Pi-hole.md b/docs/Infrastructure/Pi-hole.md index a39bd85..68bcec0 100644 --- a/docs/Infrastructure/Pi-hole.md +++ b/docs/Infrastructure/Pi-hole.md @@ -50,7 +50,7 @@ The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `D ## Host prerequisites -1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any later-sorted drop-in that resets the value in the home trash, and applies it with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** a second drop-in (`99-rootless-ports.conf`, sorted after) reset the value to 80. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until the conflict was removed. +1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any drop-in that sets a value above 53 in the home trash, comments out a stale active line in `/etc/sysctl.conf`, and applies everything with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** the value was reset to 80 by *two* sources — a later-sorted drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl processes **after** all of `sysctl.d/` and therefore overrides every drop-in. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until both conflicts were removed. 2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns. Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them. @@ -63,7 +63,7 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them. | A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] | | The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit | | Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong | -| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: exactly one file under `/etc/sysctl.d/` may set `ip_unprivileged_port_start`, and it must say 53. Redeploy pihole — `deploy_pihole` reinstalls the drop-in and neutralizes conflicts | +| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line | | Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 `), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client | ## See Also diff --git a/docs/log.md b/docs/log.md index f017be0..e7127a8 100644 --- a/docs/log.md +++ b/docs/log.md @@ -30,10 +30,10 @@ timestamp: 2026-09-14 - `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment. - `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion. - `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages. -- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): a later-sorted sysctl drop-in (`99-rootless-ports.conf`) reset `net.ipv4.ip_unprivileged_port_start` to 80, so the rootless port-53 publish failed with `bind: permission denied`. +- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too). - `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed. - `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`. -- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting later-sorted drop-ins in the home trash, and applies `sysctl --system` (the boot code path). +- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path). - `/scripts/deploy-remote.sh` — `deploy_caddy` installs/enables/starts the quadlet (retiring the bare container; all state is in host volumes); `deploy_pihole` enforces the sysctl prerequisite. - `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet. - `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode. diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index b89f41f..5440d38 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -449,16 +449,28 @@ deploy_open_webui() { deploy_pihole() { # Rootless port 53 needs net.ipv4.ip_unprivileged_port_start <= 53, and it - # must hold at boot (systemd-sysctl applies /etc/sysctl.d). The repo - # drop-in is the single source of truth; a later-sorted file that resets - # the value to 80 took LAN DNS down after the 2026-08-26 reboot, so stage - # any such conflict in the home trash instead of deleting it. + # must hold at boot. The repo drop-in is the single source of truth. + # systemd-sysctl applies /etc/sysctl.d/*.conf (lexicographic), then + # /run/sysctl.d, /usr/lib/sysctl.d, and finally /etc/sysctl.conf — later + # files override earlier ones. On 2026-08-26 a later-sorted drop-in (=80) + # AND a stale line in /etc/sysctl.conf (=80) both reset the value after a + # reboot and took LAN DNS down. Neutralize every conflicting source: + # stage sysctl.d files that set a value above 53 in the home trash, and + # comment out an active line in /etc/sysctl.conf (idempotent). run_privileged install -m 0644 dankclaw/pihole/sysctl/99-dnsmasq.conf \ /etc/sysctl.d/99-dnsmasq.conf - if grep -qs 'ip_unprivileged_port_start' /etc/sysctl.d/99-rootless-ports.conf; then - local sysctl_trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-sysctl-rootless-ports" - mkdir -p "$sysctl_trash" - run_privileged mv /etc/sysctl.d/99-rootless-ports.conf "$sysctl_trash/" + local sysctl_file sysctl_val sysctl_trash + while IFS= read -r sysctl_file; do + [[ "$sysctl_file" == */99-dnsmasq.conf ]] && continue + sysctl_val="$(sed -nE 's/^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=[[:space:]]*([0-9]+).*/\1/p' "$sysctl_file" | head -1)" + if [[ "$sysctl_val" =~ ^[0-9]+$ ]] && (( sysctl_val > 53 )); then + sysctl_trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-sysctl-unprivileged-port" + mkdir -p "$sysctl_trash" + run_privileged mv "$sysctl_file" "$sysctl_trash/" + fi + done < <(grep -lE '^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=' /etc/sysctl.d/*.conf 2>/dev/null || true) + if [[ -f /etc/sysctl.conf ]] && grep -qE '^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=' /etc/sysctl.conf; then + run_privileged sed -i -E 's/^[[:space:]]*(net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=)/# \1 managed by dankclaw\/pihole\/sysctl\/99-dnsmasq.conf/' /etc/sysctl.conf fi run_privileged sysctl --system >/dev/null mkdir -p /home/dank/Containers/pihole/etc-pihole /home/dank/Containers/pihole/etc-dockers diff --git a/tests/pihole.bats b/tests/pihole.bats index 7fc395d..60b9987 100644 --- a/tests/pihole.bats +++ b/tests/pihole.bats @@ -62,6 +62,12 @@ HOSTS="$PIHOLE_DIR/addn-hosts" grep -q 'sysctl --system' "$REPO_ROOT/scripts/deploy-remote.sh" } +@test "deploy_pihole neutralizes the sysctl.conf override (processed after sysctl.d)" { + # systemd-sysctl applies /etc/sysctl.conf LAST, so a stale line there + # beats every drop-in. The 2026-08-26 port-53 outage had one. + grep -q '/etc/sysctl.conf' "$REPO_ROOT/scripts/deploy-remote.sh" +} + @test "the pihole env example has placeholder values only" { [ -f "$ENV_EXAMPLE" ] grep -q '^WEBPASSWORD=changeme$' "$ENV_EXAMPLE"