diff --git a/AGENTS.md b/AGENTS.md index 4230a64..6693f99 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -130,6 +130,43 @@ graph LR G --> H[Per-service health check] ``` +## Boot Persistence & Self-Healing (MANDATORY) + +Every Podman container governed by this project that is successfully deployed +and proven working MUST have a self-healing, boot-persistent Quadlet unit. A +container that runs only from `compose.yaml` (or a manual `podman run`) dies +silently at reboot: rootless Podman does not auto-start bare containers, and +`restart: unless-stopped` does not survive the explicit stop that shutdown +performs. Do not report such a service as "up" without its unit. + +Each container's unit must provide: + +- **Boot persistence** — `[Install] WantedBy=default.target`, installed to + `~/.config/containers/systemd/.container` (generating + `.service`) and enabled in the user manager + (`systemctl --user enable .service`). Linger must be on for the deploy + user (`loginctl show-user dank -p Linger` → `Linger=yes`) so user units start + at boot with no login session. +- **Self-healing** — `[Service] Restart=always` (or `Restart=on-failure`) with + a short `RestartSec`, so a crash or OOM brings the container back without + operator action. Add a quadlet health check where the service has one. +- **Repo provenance** — the `.container` file lives in this repo + (`dankclaw//quadlet/` or `dankserver//quadlet/`), is installed by + the service-scoped deployer, and is covered by the quadlet test seam. The + host-side unit is generated state; the repo file is the source of truth. + +Verification before a service is declared deployed: +`systemctl --user is-enabled .service` → `enabled`, +`systemctl --user is-active .service` → `active`, and the container +answers its health check. A running container with no unit is a gap: write the +unit, test it, deploy it. + +Incident that established this rule (2026-08-26, Dankclaw reboot): Caddy ran +for weeks as a bare compose container and never came back after reboot; the +Pi-hole unit existed but crash-looped on a sysctl conflict (a later-sorted +drop-in reset `net.ipv4.ip_unprivileged_port_start` to 80), taking LAN DNS +down with it. + ## Testing Tests are mandatory before commit. `bats` must be installed on the @@ -409,7 +446,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history. | Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. | | Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. | | Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | -| Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). | +| Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). | | Container without a Quadlet unit | **Give it one** — self-healing, boot-persistent `.container` unit via git (see Boot Persistence & Self-Healing). | | Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. | | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. | | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |