alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line
This commit is contained in:
Frank committed 2026-09-17 13:09:59 -07:00
1 parent 17450dd172
commit 590853c133
17 files changed
+513 -51

No files matched your search

+12 -9
View File
@@ -14,12 +14,13 @@
# LlmSpecDecodeOff takes its place: the failure an operator actually
# cannot see from the outside on this deployment.
#
# IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
# firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
# the LLM fleet dashboard graphs) and nowhere else. Turning these into something
# that reaches a person means adding an Alertmanager or using Grafana's own
# alerting with a contact point — a decision about how Frank wants to be paged,
# not a missing line of configuration.
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
# alertmanager.container), whose single receiver webhooks the host-side
# signal-relay user unit, which hands the text to signal-cli under the
# existing account on this host. Before that date a firing alert appeared on
# Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
# dashboard graphs) and nowhere else.
groups:
- name: llm-fleet
interval: 30s
@@ -77,9 +78,11 @@ groups:
summary: "{{ $labels.job }}: serving without speculative decoding"
description: >-
Tokens are being produced but no draft has been proposed for 15
minutes. On frank-medium that means the DFlash2 drafter is not
running (check the unit's log for common_speculative_impl_draft_dflash),
and the endpoint is decoding autoregressively.
minutes. On the llama.cpp endpoints that means the drafter is not
running (check the unit's log for the
"adding speculative implementation" line — draft-mtp on both
endpoints as of 2026-09-17), and the endpoint is decoding
autoregressively.
# The endpoint, not a replica: one process serves each port now, so a
# restart of the unit is a scrape gap, not a failover.