Files

T

Frank 590853c133 alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line

2026-09-17 13:09:59 -07:00

4.1 KiB

Raw Blame History

Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their /metrics endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy
Data	`/opt/prometheus/data` (TSDB)
Retention	`--storage.tsdb.retention.time=30d`
Alerting	5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`, `node-dankserver`	`192.168.0.143:9100`, `192.168.0.235:9100`	host metrics from each machine's Node-Exporter
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-medium`, `frank-large`	`192.168.0.235:7770`, `192.168.0.143:7770`	the LLM fleet — `vllm:` and `llamacpp:` series (LAN addresses, not the guarded vhosts)

Three jobs read down, and they are not outages (audited 2026-09-16):

Job	Target	Why it never reports up
`comfyui`	`comfyui:8188`	scrapes `/`, which is HTML — "unsupported Content-Type"
`services`	`192.168.0.143:8081`, `:3005`	same: up/down intent, implemented as a scrape
`gitea`	`192.168.0.143:3000/metrics`	Gitea's exporter is off until `[metrics] ENABLED = true`

A scrape cannot express "is this HTTP service answering?" — that is what a probe is for, and the remedy is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives the guarded vhosts a real uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config`
`/prometheus`	`/opt/prometheus/data`

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on.

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes 192.168.0.143:9100, which is exactly the LAN address node-exporter.container binds (Network=host, --web.listen-address twice). Change one without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats seam pins the pairing.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here).
The lifecycle endpoint is enabled (--web.enable-lifecycle), which is why this API must never be published beyond loopback.

4.1 KiB Raw Blame History