alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line
This commit is contained in:
Frank committed 2026-09-17 13:09:59 -07:00
1 parent 17450dd172
commit 590853c133
17 files changed
+513 -51

No files matched your search

+19 -9
View File
@@ -6,12 +6,21 @@ global:
scrape_timeout: 10s
evaluation_interval: 15s
# Alerting and recording rules for the LLM fleet. They evaluate and expose state
# through /alerts, the ALERTS metric and the Grafana dashboard; nothing is
# notified, because this stack has no Alertmanager (see the rules file header).
# Alerting and recording rules for the LLM fleet. They evaluate, expose
# state through /alerts and the ALERTS metric, and — since 2026-09-17 —
# deliver to Signal through Alertmanager (see the alerting section below).
rule_files:
- /etc/prometheus/rules/*.yml
# The delivery half. Alertmanager runs on the same dankclaw_default network,
# so it is addressed by container name, like grafana and loki below. Its
# single receiver webhooks the host-side signal-relay (loopback :18093,
# reachable from the container only as host.containers.internal).
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
scrape_configs:
# Prometheus itself
- job_name: "prometheus"
@@ -62,12 +71,13 @@ scrape_configs:
# LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
# machine clients and Prometheus cannot solve an Anubis challenge.
# frank-medium is vLLM, and its series are the ones that explain behaviour
# under load — vllm:kv_cache_usage_perc (how full the per-card pool is),
# vllm:num_requests_waiting (queue depth), the request-latency histograms,
# and the prefix-cache pair vllm:prefix_cache_hits_total over
# vllm:prefix_cache_queries_total (their rates give the hit rate, which says
# whether pinned conversations are actually being reused).
# Both LLM endpoints are llama.cpp, so both export the same llamacpp:
# counter family: prompt/decode throughput, requests_deferred (queue depth
# — the only form llama.cpp exposes the capacity wall in), the
# spec_decode_num_drafts / spec_decode_num_accepted pair (proof the drafter
# is running), and the prefix-cache pair prompt_tokens_cached_total over
# prompt_tokens_total (their rates are the reuse rate, which says whether
# pinned conversations are actually being reused).
# frank-large is deliberately NOT scraped: the halogen engine that serves it
# publishes /health, /cache and /v1/* and no Prometheus text endpoint, so a
# target here would be red by construction and would fire LlmEndpointDown