dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

Browse Source 
- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line
This commit is contained in:
Frank
2026-09-17 13:09:59 -07:00
parent 17450dd172
commit 590853c133
17 changed files with 513 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Alertmanager.md
+50
View File
@@ -0,0 +1,50 @@
# Alertmanager
Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
**Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/prom/alertmanager:latest` |
| Published | `127.0.0.1:9093` → 9093, loopback only (see [[Observability]] → Exposure) |
| Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
| Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
| Data | `/opt/alertmanager/data` (silences, notification log) |
| Health | `wget --spider -q http://localhost:9093/-/healthy` |
| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` |
| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h |
## Why the webhook address is `host.containers.internal`
The Signal relay is a host-side process on the host's **loopback** (127.0.0.1:18093). From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — so the LAN IP and the podman gateway both refuse an unpublished host bind. `host.containers.internal` (169.254.1.2) is the one address that reaches a host-local listener from a bridge container. The relay listens on loopback only, so the unauthenticated webhook endpoint stays off every host interface.
## The Signal relay
`scripts/signal-relay.py` is a systemd **user** unit (`scripts/signal-relay.service`), not a Quadlet: it runs `python3` from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be `enable`d (generated quadlet units cannot on this host). Linger is on for `dank`, so `WantedBy=default.target` is the whole boot story.
- POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
- GET `/health` — liveness for the deployer's health wait.
- Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
- Sends through `/usr/local/bin/signal-cli -a 318311 send +12062476336` — Frank's existing account; the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
## Deployment
```bash
./scripts/deploy.sh dankclaw --service alertmanager
```
`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, enables and restarts it, then waits on `127.0.0.1:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.
## Gotchas
- The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
- The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
- signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
## See Also
- [[Prometheus]] — the rule evaluator that ships the alerts
- [[Observability]] — the full stack
serverAppConfigs-docs/Infrastructure/Observability.md
+13 -10
View File
@@ -1,14 +1,14 @@
---
type: reference
title: Observability on Dankclaw
description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured.
description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured.
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11
---
# Observability
The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.
The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards, node-exporter for host metrics, and Alertmanager for alert delivery — to Signal, through a host-side relay. This is a small homelab — no Kubernetes.
## Stack
@@ -21,7 +21,9 @@ graph LR
A --> L[Loki]
P --> G[Grafana]
L --> G
```
P -->|alerts| AM[Alertmanager]
AM -->|webhook| R[Signal relay]
R -->|signal-cli| SIG[Signal]
| Component | Role | Image | Published |
|-----------|------|-------|-----------|
@@ -30,6 +32,7 @@ graph LR
| [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
| [[Alertmanager]] | alert grouping & delivery | `prom/alertmanager` | `127.0.0.1:9093` |
## Current state: installed, healthy, and mostly unconfigured
@@ -37,17 +40,17 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
- **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard'…
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the netavark host-gateway address, because the relay listens on the host's loopback and bridge containers cannot reach a host bind at the LAN IP. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.
## Deployment
Six units now, across two hosts. Dankclaw runs five (Prometheus, Grafana, Loki,
Alloy, node-exporter) from `dankclaw/observability/quadlet/`; **Dankserver runs
Seven units now, across two hosts. Dankclaw runs six (Prometheus, Grafana, Loki,
Alloy, node-exporter, Alertmanager) from `dankclaw/observability/quadlet/`; **Dankserver runs
its own node-exporter** from `dankserver/observability/quadlet/`, added
2026-09-16 after the same reboot argument took it down a second time — it had
been a compose-only service, died `Exited (143)` at a reboot, and left
@@ -58,7 +61,7 @@ compose form also bound the wildcard address). The deployer installs from
The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**
```bash
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter|alertmanager
./scripts/deploy.sh dankserver --service node-exporter
```
@@ -70,7 +73,7 @@ Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podm
The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:
- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
- **Prometheus, Grafana, Loki, Alloy and Alertmanager publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
- **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.
## Scrape surface
@@ -93,7 +96,7 @@ frank-medium by hand, and re-cut the same day when Dankclaw split into two
endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
pair that proves the DFlash2 drafter is running. What those numbers mean on
pair that proves the drafter is running (draft-mtp on both endpoints as of 2026-09-17).
this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
family on frank-small is read against [[frank-small-qwen38-gsq-rco]].
serverAppConfigs-docs/Infrastructure/Prometheus.md
+1 -1
View File
@@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
| Data | `/opt/prometheus/data` (TSDB) |
| Retention | `--storage.tsdb.retention.time=30d` |
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) |
| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see [[Alertmanager]]) |
## Targets