Frank
d251dbfe62
The bridge-subnet-only pair dropped the host's own health probe (source 192.168.0.143/127.0.0.1) and failed a healthy deploy. The deployer now installs the full set: ACCEPT 127.0.0.1, ACCEPT the host IP, ACCEPT 10.89.0.0/16, then DROP.
51 lines
4.5 KiB
Markdown
51 lines
4.5 KiB
Markdown
# Alertmanager
|
|
|
|
Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
|
|
|
|
**Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/prom/alertmanager:latest` |
|
|
| Published | `127.0.0.1:9093` → 9093, loopback only (see [[Observability]] → Exposure) |
|
|
| Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
|
|
| Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
|
|
| Data | `/opt/alertmanager/data` (silences, notification log) |
|
|
| Health | `wget --spider -q http://localhost:9093/-/healthy` |
|
|
| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` |
|
|
| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h |
|
|
|
|
## Why the webhook address is `host.containers.internal`
|
|
|
|
The Signal relay is a host-side process, and the address it must bind is the one detail this design gets wrong twice (2026-09-17). From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — so a relay bound to 127.0.0.1 refused the first webhook with `dial tcp 169.254.1.2:18093: connection refused`. The obvious fix — bind `host.containers.internal` itself — fails too: on this host netavark runs its bridges in aardvark-dns' netns, so 169.254.1.2 is not a local address in the host's namespace (`OSError: [Errno 99] Cannot assign requested address`). What works: bind the **host's LAN IP** (192.168.0.143) — netavark delivers the container's `169.254.1.2` dial to that address (proven with a probe socket: container → `169.254.1.2:18094` → host socket bound to `192.168.0.143:18094`, while the LAN IP refused from the container, the usual netavark quirk). Since that puts an unauthenticated endpoint on a host interface, the deployer installs an iptables INPUT set admitting 18093/tcp from loopback, the host itself (its own health probe) and the podman bridge subnet (10.89.0.0/16) only — the rest of the LAN gets a DROP.
|
|
|
|
## The Signal relay
|
|
|
|
`scripts/signal-relay.py` is a systemd **user** unit (`scripts/signal-relay.service`), not a Quadlet: it runs `python3` from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be `enable`d (generated quadlet units cannot on this host). Linger is on for `dank`, so `WantedBy=default.target` is the whole boot story.
|
|
|
|
- POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
|
|
- GET `/health` — liveness for the deployer's health wait.
|
|
- Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
|
|
- Sends through `/usr/local/bin/signal-cli -a 318311 send +12062476336` — Frank's existing account; the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
./scripts/deploy.sh dankclaw --service alertmanager
|
|
```
|
|
|
|
`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, installs the iptables rules, enables and restarts the relay, then waits on `192.168.0.143:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.
|
|
|
|
## Gotchas
|
|
|
|
- The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
|
|
- The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
|
|
- signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
|
|
|
|
## See Also
|
|
|
|
- [[Prometheus]] — the rule evaluator that ships the alerts
|
|
- [[Observability]] — the full stack
|