alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line
This commit is contained in:
Frank committed 2026-09-17 13:09:59 -07:00
1 parent 17450dd172
commit 590853c133
17 files changed
+513 -51

No files matched your search

+11 -8
View File
@@ -122,14 +122,17 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured as of 2026-09-16**: a
Grafana datasource and the `llm-fleet` dashboard are provisioned from
`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
Alertmanager, so a firing alert reaches nobody (an operator decision, not an
oversight). Five Prometheus targets read down for structural reasons
(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
See `serverAppConfigs-docs/Infrastructure/Observability.md`.
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, loopback :18093, reached from the
container as `host.containers.internal`) → signal-cli account 318311. Five
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.
## Testing