alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone
- dankclaw/observability/quadlet/alertmanager.container: loopback :9093, config from shared/prometheus/alertmanager.yml, family conventions - shared/prometheus/alertmanager.yml: single signal receiver, group_by alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093 - scripts/signal-relay.py + .service: host-side user unit, loopback :18093, 15-min dedupe, sends via signal-cli account 318311 - prometheus.yml: alerting block to alertmanager:9093; stale comments fixed - llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic - deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config copy, relay install, planner routing for config + relay files - homepage: alertmanager status card (loopback probe) - tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new delivery-path seams (webhook<->relay contract, prometheus target, relay unit, signal-cli account, planner routing) - docs: Alertmanager.md component page, Observability.md, Prometheus.md, index.md, AGENTS.md delivery line
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
# Quadlet unit: alertmanager (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/alertmanager.container
|
||||
# Unit name: alertmanager.service (container name: alertmanager)
|
||||
#
|
||||
# The delivery half of the LLM alert rules. Until 2026-09-17 the rules in
|
||||
# shared/prometheus/rules/llm-fleet.yml evaluated and reached nobody — a
|
||||
# firing alert was visible in Prometheus and on the Grafana dashboard and
|
||||
# nowhere else, and an undelivered alert looks identical to a quiet one.
|
||||
# This unit closes that gap: Prometheus pushes to alertmanager:9093 over the
|
||||
# dankclaw_default network, and the single receiver webhooks the host-side
|
||||
# signal-relay user unit, which hands the text to signal-cli under Frank's
|
||||
# existing Signal account.
|
||||
#
|
||||
# PUBLISHING. 127.0.0.1 only, like every other observability unit: the
|
||||
# unauthenticated Alertmanager API (with the lifecycle endpoint) has no
|
||||
# business on a host interface, and no off-host consumer needs it. Caddy
|
||||
# gains no vhost for it; the homepage timer probes loopback.
|
||||
#
|
||||
# CONFIG. /opt/alertmanager/config/alertmanager.yml is installed from
|
||||
# shared/prometheus/alertmanager.yml by deploy-remote.sh on every deploy, so
|
||||
# the repo copy is the source of truth. STATE lives in /opt/alertmanager/data.
|
||||
#
|
||||
# Rootless: container root maps to the invoking user (dank) in this user
|
||||
# namespace and dank owns /opt/alertmanager/data. The image's default user
|
||||
# (nobody) maps to a subuid that cannot write it — same User=0:0 as prometheus.
|
||||
|
||||
[Unit]
|
||||
Description=Alertmanager alert delivery (Dankclaw)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=alertmanager
|
||||
Image=docker.io/prom/alertmanager:latest
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.enable-lifecycle
|
||||
PublishPort=127.0.0.1:9093:9093
|
||||
Volume=/opt/alertmanager/config:/etc/alertmanager
|
||||
Volume=/opt/alertmanager/data:/alertmanager
|
||||
HealthCmd=wget --spider -q http://localhost:9093/-/healthy
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -27,10 +27,11 @@
|
||||
#
|
||||
# RULES. Five LLM alert rules and one recording rule load from
|
||||
# /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
|
||||
# deployer). There is still **no Alertmanager**, so nothing is delivered: a
|
||||
# firing alert is visible in Prometheus and on the Grafana dashboard's ALERTS
|
||||
# panel and reaches nobody. Delivery is an operator decision, not an oversight;
|
||||
# the fragility is that an undelivered alert looks identical to a quiet one.
|
||||
# deployer). Delivery went live 2026-09-17: the alerting block in
|
||||
# prometheus.yml points at alertmanager:9093, which webhooks every firing
|
||||
# group to the host-side Signal relay (127.0.0.1:18093, reached as
|
||||
# host.containers.internal), which sends through signal-cli account 318311.
|
||||
# The Grafana dashboard's ALERTS panel remains the second surface.
|
||||
#
|
||||
# The config volume is also why rules belong in git: editing them in the UI is
|
||||
# lost on the next deploy, since the deployer overwrites the whole directory.
|
||||
|
||||
Reference in New Issue
Block a user