serverAppConfigs/serverAppConfigs-docs/Infrastructure/Alertmanager.md

Alertmanager

Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.

Host: Dankclaw — Port: 9093 (loopback) — Docs: https://prometheus.io/docs/alerting/latest/alertmanager/

Facts

Item	Value
Image	docker.io/prom/alertmanager:latest
Published	127.0.0.1:9093 → 9093, loopback only (see Observability → Exposure)
Unit	dankclaw/observability/quadlet/alertmanager.container → alertmanager.service
Config	/opt/alertmanager/config/alertmanager.yml, installed from shared/prometheus/alertmanager.yml on every deploy
Data	/opt/alertmanager/data (silences, notification log)
Health	wget --spider -q http://localhost:9093/-/healthy
Receiver	one signal receiver → webhook http://host.containers.internal:18093/alertmanager
Grouping	group_by: [alertname], group_wait 30 s, group_interval 5 m, repeat_interval 4 h

Why the webhook address is host.containers.internal

The Signal relay is a host-side process, and the address it must bind is the one detail this design gets wrong once: 127.0.0.1 does not work. From a bridge container, loopback is the container's own, and netavark DNATs published ports only — the LAN IP and the podman gateway are both refused for an unpublished host bind. host.containers.internal (169.254.1.2) is the one address that reaches a host-local listener from a bridge container, so the relay binds that address (measured 2026-09-17: a loopback bind refused the first webhook with dial tcp 169.254.1.2:18093: connection refused). 169.254.1.2 is link-local to the podman bridge — not on loopback, not on any LAN interface, reachable only from this host's containers — so the unauthenticated webhook endpoint stays off the wire.

The Signal relay

scripts/signal-relay.py is a systemd user unit (scripts/signal-relay.service), not a Quadlet: it runs python3 from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be enabled (generated quadlet units cannot on this host). Linger is on for dank, so WantedBy=default.target is the whole boot story.

• POST /alertmanager — the Alertmanager webhook handler; formats the group and sends it.
• GET /health — liveness for the deployer's health wait.
• Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h repeat_interval during a relay hiccup.
• Sends through /usr/local/bin/signal-cli -a 318311 send +12062476336 — Frank's existing account; the credential is the account state under ~/.local/share/signal-cli, and the account/phone are numbers, not secrets.

Deployment

./scripts/deploy.sh dankclaw --service alertmanager

deploy_observability installs the quadlet unit, copy_alertmanager_config installs the config, and install_signal_relay copies the user unit, enables and restarts it, then waits on 169.254.1.2:18093/health before the container's own /-/healthy wait. The planner selects alertmanager for its quadlet file, shared/prometheus/alertmanager.yml, and the two relay files; prometheus ships the alerting: block that points at alertmanager:9093.

Gotchas

• The webhook URL and the relay's listen port are a cross-file contract: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. tests/observability_quadlet.bats pins both halves.
• The container address is alertmanager:9093, never the published port: Prometheus is in a container and the port is loopback-only.
• signal-cli account state is on the host, not in the container — a wiped ~/.local/share/signal-cli silences delivery while every status page stays green.

See Also

• Prometheus — the rule evaluator that ships the alerts
• Observability — the full stack