alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone

- dankclaw/observability/quadlet/alertmanager.container: loopback :9093, config from shared/prometheus/alertmanager.yml, family conventions - shared/prometheus/alertmanager.yml: single signal receiver, group_by alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093 - scripts/signal-relay.py + .service: host-side user unit, loopback :18093, 15-min dedupe, sends via signal-cli account 318311 - prometheus.yml: alerting block to alertmanager:9093; stale comments fixed - llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic - deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config copy, relay install, planner routing for config + relay files - homepage: alertmanager status card (loopback probe) - tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new delivery-path seams (webhook<->relay contract, prometheus target, relay unit, signal-cli account, planner routing) - docs: Alertmanager.md component page, Observability.md, Prometheus.md, index.md, AGENTS.md delivery line
2026-09-17 13:09:59 -07:00
parent 17450dd172
commit 590853c133
17 changed files with 513 additions and 51 deletions
@@ -122,14 +122,17 @@ This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
 bare compose container never returned; 2026-09-11: five observability
 services had no units). The observability services now have units under
 `dankclaw/observability/quadlet/` and their compose definitions were removed —
-one owner per container name. The stack is **configured as of 2026-09-16**: a
-Grafana datasource and the `llm-fleet` dashboard are provisioned from
-`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
-loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
-Alertmanager, so a firing alert reaches nobody (an operator decision, not an
-oversight). Five Prometheus targets read down for structural reasons
-(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
-See `serverAppConfigs-docs/Infrastructure/Observability.md`.
+one owner per container name. The stack is **configured and delivering as of
+2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
+provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
+llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
+delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
+alertmanager.container`, loopback :9093) → host-side Signal relay
+(`scripts/signal-relay.py`, user unit, loopback :18093, reached from the
+container as `host.containers.internal`) → signal-cli account 318311. Five
+Prometheus targets read down for structural reasons (HTML-scraped "probes",
+Gitea metrics off); the remedy is a blackbox exporter. See
+`serverAppConfigs-docs/Infrastructure/Observability.md`.

 ## Testing

@@ -0,0 +1,53 @@
+# Quadlet unit: alertmanager (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/alertmanager.container
+# Unit name: alertmanager.service (container name: alertmanager)
+#
+# The delivery half of the LLM alert rules. Until 2026-09-17 the rules in
+# shared/prometheus/rules/llm-fleet.yml evaluated and reached nobody — a
+# firing alert was visible in Prometheus and on the Grafana dashboard and
+# nowhere else, and an undelivered alert looks identical to a quiet one.
+# This unit closes that gap: Prometheus pushes to alertmanager:9093 over the
+# dankclaw_default network, and the single receiver webhooks the host-side
+# signal-relay user unit, which hands the text to signal-cli under Frank's
+# existing Signal account.
+#
+# PUBLISHING. 127.0.0.1 only, like every other observability unit: the
+# unauthenticated Alertmanager API (with the lifecycle endpoint) has no
+# business on a host interface, and no off-host consumer needs it. Caddy
+# gains no vhost for it; the homepage timer probes loopback.
+#
+# CONFIG. /opt/alertmanager/config/alertmanager.yml is installed from
+# shared/prometheus/alertmanager.yml by deploy-remote.sh on every deploy, so
+# the repo copy is the source of truth. STATE lives in /opt/alertmanager/data.
+#
+# Rootless: container root maps to the invoking user (dank) in this user
+# namespace and dank owns /opt/alertmanager/data. The image's default user
+# (nobody) maps to a subuid that cannot write it — same User=0:0 as prometheus.
+
+[Unit]
+Description=Alertmanager alert delivery (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=alertmanager
+Image=docker.io/prom/alertmanager:latest
+Network=dankclaw_default
+User=0:0
+Exec=--config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.enable-lifecycle
+PublishPort=127.0.0.1:9093:9093
+Volume=/opt/alertmanager/config:/etc/alertmanager
+Volume=/opt/alertmanager/data:/alertmanager
+HealthCmd=wget --spider -q http://localhost:9093/-/healthy
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -27,10 +27,11 @@
 #
 # RULES. Five LLM alert rules and one recording rule load from
 # /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
-# deployer). There is still **no Alertmanager**, so nothing is delivered: a
-# firing alert is visible in Prometheus and on the Grafana dashboard's ALERTS
-# panel and reaches nobody. Delivery is an operator decision, not an oversight;
-# the fragility is that an undelivered alert looks identical to a quiet one.
+# deployer). Delivery went live 2026-09-17: the alerting block in
+# prometheus.yml points at alertmanager:9093, which webhooks every firing
+# group to the host-side Signal relay (127.0.0.1:18093, reached as
+# host.containers.internal), which sends through signal-cli account 318311.
+# The Grafana dashboard's ALERTS panel remains the second surface.
 #
 # The config volume is also why rules belong in git: editing them in the UI is
 # lost on the next deploy, since the deployer overwrites the whole directory.
@@ -38,6 +38,7 @@ SERVICE_ORDER = (
    "loki",
    "alloy",
    "node-exporter",
+    "alertmanager",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -83,6 +84,7 @@ DANKCLAW_SERVICES = {
    "loki",
    "alloy",
    "node-exporter",
+    "alertmanager",
    "authentik-postgres",
    "authentik-server",
    "authentik-worker",
@@ -246,6 +248,10 @@ def services_for_path(path: str, target: str) -> set[str]:
            _add(services, "searxng")
        elif group == "prometheus":
            _add(services, "prometheus")
+            if len(parts) >= 3 and parts[2] == "alertmanager.yml":
+                # The Alertmanager config ships from this directory: a change
+                # to it must redeploy the service that installs it.
+                _add(services, "alertmanager")
        elif group == "loki":
            _add(services, "loki")
        elif group == "alloy":
@@ -320,7 +326,7 @@ def services_for_path(path: str, target: str) -> set[str]:
            # consumer (prometheus before grafana, loki before alloy).
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
                    _add(services, name)
            return services
        if len(parts) >= 2 and parts[1] == "gitea":
@@ -391,6 +397,11 @@ def services_for_path(path: str, target: str) -> set[str]:
        name = p.name
        if name in {"homepage-status.py", "homepage-status.service", "homepage-status.timer"}:
            _add(services, "homepage")
+        if name in {"signal-relay.py", "signal-relay.service"}:
+            # The Signal relay is installed by the alertmanager deploy, so a
+            # change to it redeploying alertmanager is the only path that
+            # reaches it on the host.
+            _add(services, "alertmanager")
        if name in {"frank-small.env", "frank-small.service", "serve-frank-small.sh"}:
            # llama.cpp frank-small (2026-09-16) keeps its unit, env and serve
            # files here; each one selects frank-small only. The ik_llama.cpp
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -210,6 +210,31 @@ copy_prometheus_config() {
        /opt/prometheus/config/rules/llm-fleet.yml
 }

+# Alertmanager config ships beside the Prometheus config: the two halves of
+# the alerting path change together, and a receiver with no route (or a
+# route with no receiver) fails silently.
+copy_alertmanager_config() {
+    run_privileged mkdir -p /opt/alertmanager/config /opt/alertmanager/data
+    run_privileged cp shared/prometheus/alertmanager.yml /opt/alertmanager/config/alertmanager.yml
+}
+
+# The Signal relay is Alertmanager's only webhook consumer and the only
+# thing on this host that talks to signal-cli. It is a plain systemd USER
+# unit (not a quadlet): it runs python3 from the checkout, so a deploy
+# updates it without a copy step, and — unlike generated quadlet units,
+# which this host's podman 5.x v1 generator rejects — a hand-written user
+# unit can be enabled, and Linger=yes for dank starts it at boot.
+install_signal_relay() {
+    local unit_dir="$HOME/.config/systemd/user"
+    mkdir -p "$unit_dir"
+    cp scripts/signal-relay.service "$unit_dir/signal-relay.service"
+    systemctl --user daemon-reload
+    systemctl --user enable signal-relay.service
+    systemctl --user restart signal-relay.service
+    curl -fsS -m 5 http://127.0.0.1:18093/health >/dev/null \
+        || { echo "signal-relay is not answering on 127.0.0.1:18093" >&2; return 1; }
+}
+
 copy_loki_config() {
    run_privileged mkdir -p /opt/loki/config
    run_privileged cp shared/loki/loki-config.yaml /opt/loki/config/loki-config.yaml
@@ -426,13 +451,17 @@ seed_grafana_env() {
 deploy_observability() {
    local service="$1"
    # Observability units live under <target>/observability/quadlet: Dankclaw
-    # has five, Dankserver one (node-exporter, since 2026-09-16). The config
+    # has six, Dankserver one (node-exporter, since 2026-09-16). The config
    # copies below are Dankclaw-owned state and run only there.
    local unit_dir="$TARGET/observability/quadlet"
    case "$service" in
        prometheus)
            copy_prometheus_config
            ;;
+        alertmanager)
+            copy_alertmanager_config
+            install_signal_relay
+            ;;
        loki)
            copy_loki_config
            build_compose_image loki
@@ -456,6 +485,7 @@ deploy_observability() {
    systemctl --user restart "$service.service"
    case "$service" in
        prometheus) wait_for_health prometheus http://127.0.0.1:9090/-/healthy 60 ;;
+        alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
        grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
        alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
@@ -1378,7 +1408,7 @@ deploy_service() {
        copyparty) deploy_copyparty ;;
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
-        prometheus|grafana|loki|alloy|node-exporter) deploy_observability "$service" ;;
+        prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -138,6 +138,7 @@ valid_service() {
        dankclaw:gitea-server|dankclaw:homepage|\
        dankclaw:comfyui|dankclaw:hermes|dankclaw:hermes-dashboard|\
        dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|dankclaw:loki|dankclaw:alloy|\
+        dankclaw:alertmanager|\
        dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
        dankclaw:frank-large|dankclaw:frank-small|\
@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+"""Alertmanager-to-Signal relay for the homelab (Dankclaw).
+
+Listens on 127.0.0.1:18093, receives Alertmanager's webhook payload, formats
+each alert group into a short message and hands it to signal-cli under the
+existing account on this host. Runs as a systemd user unit
+(scripts/signal-relay.service); logs go to the journal.
+
+The phone number and account name are numbers, not credentials — the
+credential is the signal-cli account state under
+~/.local/share/signal-cli, which never leaves the host.
+
+Stdlib only: this box's user units run the system python3.
+"""
+
+import json
+import shutil
+import subprocess
+import sys
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+LISTEN_ADDR = ("127.0.0.1", 18093)
+SIGNAL_CLI = "/usr/local/bin/signal-cli"
+SIGNAL_ACCOUNT = "318311"
+SIGNAL_TO = "+12062476336"
+
+# The relay is a safety net, not the rate limiter: Alertmanager already
+# repeats on repeat_interval. This suppresses exact duplicates that arrive
+# inside the window (config reloads, double-fires) so a misconfiguration
+# cannot fill the phone.
+DEDUPE_SECONDS = 15 * 60
+_last_sent = {}
+
+# Cap on the instances listed per group; the count is always stated.
+MAX_INSTANCES = 3
+
+
+def log(msg):
+    print(f"signal-relay: {msg}", file=sys.stderr, flush=True)
+
+
+def format_group(alerts):
+    """One message per webhook delivery (Alertmanager groups by alertname)."""
+    if not alerts:
+        return None
+    first = alerts[0]
+    name = first.get("labels", {}).get("alertname", "unknown")
+    state = "firing" if first.get("status") == "firing" else "resolved"
+    severity = first.get("labels", {}).get("severity", "info").upper()
+    lines = [f"[{severity}] {name} ({len(alerts)} {state})"]
+    for a in alerts[:MAX_INSTANCES]:
+        labels = a.get("labels", {})
+        ident = labels.get("instance") or labels.get("job") or "?"
+        summary = (a.get("annotations") or {}).get("summary")
+        line = f"  {ident}"
+        if summary:
+            line += f": {summary}"
+        lines.append(line)
+    if len(alerts) > MAX_INSTANCES:
+        lines.append(f"  … and {len(alerts) - MAX_INSTANCES} more")
+    return "\n".join(lines)
+
+
+def dedupe_key(alerts):
+    first = alerts[0]
+    labels = tuple(sorted(first.get("labels", {}).items()))
+    return (first.get("status"),) + labels
+
+
+def send_signal(text):
+    if shutil.which(SIGNAL_CLI) is None:
+        log(f"signal-cli not found at {SIGNAL_CLI}")
+        return False
+    try:
+        proc = subprocess.run(
+            [SIGNAL_CLI, "-a", SIGNAL_ACCOUNT, "-o", "json", "send", SIGNAL_TO, text],
+            capture_output=True, text=True, timeout=30,
+        )
+    except subprocess.TimeoutExpired:
+        log("signal-cli timed out after 30s")
+        return False
+    if proc.returncode != 0:
+        log(f"signal-cli failed rc={proc.returncode}: {proc.stderr.strip()[:300]}")
+        return False
+    return True
+
+
+def handle_alertmanager(payload):
+    alerts = payload.get("alerts", [])
+    text = format_group(alerts)
+    if text is None:
+        log("empty alert group, nothing sent")
+        return
+    now = time.monotonic()
+    key = dedupe_key(alerts)
+    _last_sent = {k: v for k, v in _last_sent.items() if now - v < DEDUPE_SECONDS}
+    if key in _last_sent:
+        log(f"suppressed duplicate: {key[0]}")
+        return
+    if send_signal(text):
+        _last_sent[key] = now
+        log(f"sent to {SIGNAL_TO}: {text.splitlines()[0]}")
+    else:
+        # Do not mark a failed send: the next repeat should retry.
+        log("send failed; will retry on the next delivery")
+
+
+class Handler(BaseHTTPRequestHandler):
+    def do_GET(self):
+        if self.path == "/health":
+            self.send_response(200)
+            self.end_headers()
+            self.wfile.write(b"ok")
+        else:
+            self.send_response(404)
+            self.end_headers()
+
+    def do_POST(self):
+        if self.path != "/alertmanager":
+            self.send_response(404)
+            self.end_headers()
+            return
+        length = int(self.headers.get("Content-Length", 0))
+        try:
+            payload = json.loads(self.rfile.read(length))
+        except (ValueError, UnicodeDecodeError) as e:
+            log(f"bad payload: {e}")
+            self.send_response(400)
+            self.end_headers()
+            return
+        try:
+            handle_alertmanager(payload)
+        except Exception as e:  # a relay bug must not 500 Alertmanager
+            log(f"handler error: {e!r}")
+        self.send_response(200)
+        self.end_headers()
+        self.wfile.write(b"ok")
+
+    def log_message(self, fmt, *args):
+        # Default access log is per-request noise; the journal keeps the
+        # send/suppress lines, which are the ones that matter.
+        pass
+
+
+def main():
+    server = ThreadingHTTPServer(LISTEN_ADDR, Handler)
+    log(f"listening on {'.'.join(map(str, LISTEN_ADDR))}, account {SIGNAL_ACCOUNT}")
+    server.serve_forever()
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,37 @@
+# Systemd USER unit: signal-relay (Dankclaw).
+#
+# Install path: ~/.config/systemd/user/signal-relay.service
+# Installed and enabled by install_signal_relay in scripts/deploy-remote.sh
+# (part of the alertmanager deploy).
+#
+# The webhook consumer for Alertmanager: it listens on loopback only and is
+# the only thing on this host that talks to signal-cli. It is a plain user
+# unit, not a quadlet, on purpose:
+#
+#   * it runs python3 straight from the checkout, so a deploy updates the
+#     relay without a copy step (the homepage timer copies to
+#     /usr/local/bin; this one has no root side);
+#   * generated quadlet units cannot be enabled on this host's podman 5.x
+#     v1 generator — a hand-written user unit can, and with Linger=yes for
+#     dank it starts at boot;
+#   * it needs no container: its only dependencies are /usr/bin/python3
+#     (system) and /usr/local/bin/signal-cli (host install, account state
+#     under ~/.local/share/signal-cli).
+#
+# Loopback-only for the same reason every observability port is: an
+# unauthenticated message-sending endpoint has no business on a host
+# interface. Alertmanager reaches it from its container via
+# host.containers.internal.
+
+[Unit]
+Description=Alertmanager-to-Signal relay (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/signal-relay.py
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,50 @@
+# Alertmanager
+
+Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
+
+**Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/
+
+## Facts
+
+| Item | Value |
+|------|-------|
+| Image | `docker.io/prom/alertmanager:latest` |
+| Published | `127.0.0.1:9093` → 9093, loopback only (see [[Observability]] → Exposure) |
+| Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
+| Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
+| Data | `/opt/alertmanager/data` (silences, notification log) |
+| Health | `wget --spider -q http://localhost:9093/-/healthy` |
+| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` |
+| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h |
+
+## Why the webhook address is `host.containers.internal`
+
+The Signal relay is a host-side process on the host's **loopback** (127.0.0.1:18093). From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — so the LAN IP and the podman gateway both refuse an unpublished host bind. `host.containers.internal` (169.254.1.2) is the one address that reaches a host-local listener from a bridge container. The relay listens on loopback only, so the unauthenticated webhook endpoint stays off every host interface.
+
+## The Signal relay
+
+`scripts/signal-relay.py` is a systemd **user** unit (`scripts/signal-relay.service`), not a Quadlet: it runs `python3` from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be `enable`d (generated quadlet units cannot on this host). Linger is on for `dank`, so `WantedBy=default.target` is the whole boot story.
+
+- POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
+- GET `/health` — liveness for the deployer's health wait.
+- Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
+- Sends through `/usr/local/bin/signal-cli -a 318311 send +12062476336` — Frank's existing account; the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
+
+## Deployment
+
+```bash
+./scripts/deploy.sh dankclaw --service alertmanager
+```
+
+`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, enables and restarts it, then waits on `127.0.0.1:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.
+
+## Gotchas
+
+- The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
+- The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
+- signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
+
+## See Also
+
+- [[Prometheus]] — the rule evaluator that ships the alerts
+- [[Observability]] — the full stack
@@ -1,14 +1,14 @@
 ---
 type: reference
 title: Observability on Dankclaw
-description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured.
+description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured.
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
 timestamp: 2026-09-11
 ---

 # Observability

-The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.
+The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards, node-exporter for host metrics, and Alertmanager for alert delivery — to Signal, through a host-side relay. This is a small homelab — no Kubernetes.

 ## Stack

@@ -21,7 +21,9 @@ graph LR
    A --> L[Loki]
    P --> G[Grafana]
    L --> G
-```
+    P -->|alerts| AM[Alertmanager]
+    AM -->|webhook| R[Signal relay]
+    R -->|signal-cli| SIG[Signal]

 | Component | Role | Image | Published |
 |-----------|------|-------|-----------|
@@ -30,6 +32,7 @@ graph LR
 | [[Loki]] | log storage & query | `franksdank/loki` (local build) | `127.0.0.1:3100` |
 | [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
 | [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
+| [[Alertmanager]] | alert grouping & delivery | `prom/alertmanager` | `127.0.0.1:9093` |

 ## Current state: installed, healthy, and mostly unconfigured

@@ -37,17 +40,17 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in

 - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard'…
+- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the netavark host-gateway address, because the relay listens on the host's loopback and bridge containers cannot reach a host bind at the LAN IP. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.

-The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **no rule reaches a person** — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.
+The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, five rules evaluate over it, and **a firing rule reaches Frank's phone** — Alertmanager groups and forwards to the Signal relay, which sends through signal-cli. Correlation is still manual; the alert is just no longer silent.

 ## Deployment

-Six units now, across two hosts. Dankclaw runs five (Prometheus, Grafana, Loki,
-Alloy, node-exporter) from `dankclaw/observability/quadlet/`; **Dankserver runs
+Seven units now, across two hosts. Dankclaw runs six (Prometheus, Grafana, Loki,
+Alloy, node-exporter, Alertmanager) from `dankclaw/observability/quadlet/`; **Dankserver runs
 its own node-exporter** from `dankserver/observability/quadlet/`, added
 2026-09-16 after the same reboot argument took it down a second time — it had
 been a compose-only service, died `Exited (143)` at a reboot, and left
@@ -58,7 +61,7 @@ compose form also bound the wildcard address). The deployer installs from
 The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**

 ```bash
-./scripts/deploy.sh dankclaw --service loki          # or prometheus|grafana|alloy|node-exporter
+./scripts/deploy.sh dankclaw --service loki          # or prometheus|grafana|alloy|node-exporter|alertmanager
 ./scripts/deploy.sh dankserver --service node-exporter
 ```

@@ -70,7 +73,7 @@ Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podm

 The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:

- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
+- **Prometheus, Grafana, Loki, Alloy and Alertmanager publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
 - **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.

 ## Scrape surface
@@ -93,7 +96,7 @@ frank-medium by hand, and re-cut the same day when Dankclaw split into two
 endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
 now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
 counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
-pair that proves the DFlash2 drafter is running. What those numbers mean on
+pair that proves the drafter is running (draft-mtp on both endpoints as of 2026-09-17).
 this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
 family on frank-small is read against [[frank-small-qwen38-gsq-rco]].

@@ -16,7 +16,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
 | Data | `/opt/prometheus/data` (TSDB) |
 | Retention | `--storage.tsdb.retention.time=30d` |
-| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`). Evaluated, **not delivered**: there is no Alertmanager (see [[Observability]]) |
+| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see [[Alertmanager]]) |

 ## Targets

@@ -85,9 +85,10 @@ covers both workloads).
 - [[authentik-worker]] — Authentik background worker service
 - [[Homepage]] — password-protected service index
 - [[Homepage Design Grilling]] — homepage decision log and open questions
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter: the stack, its quadlet deployment, its exposure, and what is still missing (alert delivery)
+- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
 - [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule; evaluated but not delivered — no Alertmanager)
+- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
+- [[Alertmanager]] — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
 - [[Node-Exporter]] — host metrics
 - [[Loki]] — log storage
 - [[Alloy]] — log shipping (replaces Promtail)
@@ -186,6 +186,17 @@
    "wip": false,
    "showLink": false
  },
+  {
+    "name": "alertmanager",
+    "category": "admin",
+    "description": "Alert delivery to Signal.",
+    "url": "",
+    "docs": "",
+    "probe": "127.0.0.1:9093",
+    "method": "http",
+    "wip": false,
+    "showLink": false
+  },
  {
    "name": "Caddy",
    "category": "admin",
@@ -0,0 +1,28 @@
+# Alertmanager configuration for the homelab.
+# Deployed to /opt/alertmanager/config/alertmanager.yml on Dankclaw.
+#
+# The one delivery path is Signal. The webhook lands on the host-side
+# signal-relay (a systemd user unit listening on 127.0.0.1:18093), which
+# formats the group and hands it to signal-cli under Frank's existing
+# account. There is deliberately no second receiver: a duplicate channel is
+# how an alert starts arriving twice and getting muted everywhere.
+#
+# From this container the host's loopback is reachable only as
+# host.containers.internal (169.254.1.2): netavark DNATs published ports
+# only, so the host's LAN IP and the podman gateway are both refused for an
+# unpublished port — the same quirk that 502'd the hermes upstream in the
+# Caddyfile on 2026-09-16.
+global:
+  resolve_timeout: 5m
+
+route:
+  receiver: signal
+  group_by: ['alertname']
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 4h
+
+receivers:
+  - name: signal
+    webhook_configs:
+      - url: http://host.containers.internal:18093/alertmanager
@@ -6,12 +6,21 @@ global:
  scrape_timeout: 10s
  evaluation_interval: 15s

-# Alerting and recording rules for the LLM fleet. They evaluate and expose state
-# through /alerts, the ALERTS metric and the Grafana dashboard; nothing is
-# notified, because this stack has no Alertmanager (see the rules file header).
+# Alerting and recording rules for the LLM fleet. They evaluate, expose
+# state through /alerts and the ALERTS metric, and — since 2026-09-17 —
+# deliver to Signal through Alertmanager (see the alerting section below).
 rule_files:
  - /etc/prometheus/rules/*.yml

+# The delivery half. Alertmanager runs on the same dankclaw_default network,
+# so it is addressed by container name, like grafana and loki below. Its
+# single receiver webhooks the host-side signal-relay (loopback :18093,
+# reachable from the container only as host.containers.internal).
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ["alertmanager:9093"]
+
 scrape_configs:
  # Prometheus itself
  - job_name: "prometheus"
@@ -62,12 +71,13 @@ scrape_configs:

  # LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
  # machine clients and Prometheus cannot solve an Anubis challenge.
-  # frank-medium is vLLM, and its series are the ones that explain behaviour
-  # under load — vllm:kv_cache_usage_perc (how full the per-card pool is),
-  # vllm:num_requests_waiting (queue depth), the request-latency histograms,
-  # and the prefix-cache pair vllm:prefix_cache_hits_total over
-  # vllm:prefix_cache_queries_total (their rates give the hit rate, which says
-  # whether pinned conversations are actually being reused).
+  # Both LLM endpoints are llama.cpp, so both export the same llamacpp:
+  # counter family: prompt/decode throughput, requests_deferred (queue depth
+  # — the only form llama.cpp exposes the capacity wall in), the
+  # spec_decode_num_drafts / spec_decode_num_accepted pair (proof the drafter
+  # is running), and the prefix-cache pair prompt_tokens_cached_total over
+  # prompt_tokens_total (their rates are the reuse rate, which says whether
+  # pinned conversations are actually being reused).
  # frank-large is deliberately NOT scraped: the halogen engine that serves it
  # publishes /health, /cache and /v1/* and no Prometheus text endpoint, so a
  # target here would be red by construction and would fire LlmEndpointDown
@@ -14,12 +14,13 @@
 #     LlmSpecDecodeOff takes its place: the failure an operator actually
 #     cannot see from the outside on this deployment.
 #
-# IMPORTANT: nothing is notified. There is no Alertmanager in this stack, so a
-# firing alert appears on Prometheus' /alerts page, in the ALERTS metric (which
-# the LLM fleet dashboard graphs) and nowhere else. Turning these into something
-# that reaches a person means adding an Alertmanager or using Grafana's own
-# alerting with a contact point — a decision about how Frank wants to be paged,
-# not a missing line of configuration.
+# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
+# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
+# alertmanager.container), whose single receiver webhooks the host-side
+# signal-relay user unit, which hands the text to signal-cli under the
+# existing account on this host. Before that date a firing alert appeared on
+# Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
+# dashboard graphs) and nowhere else.
 groups:
  - name: llm-fleet
    interval: 30s
@@ -77,9 +78,11 @@ groups:
          summary: "{{ $labels.job }}: serving without speculative decoding"
          description: >-
            Tokens are being produced but no draft has been proposed for 15
-            minutes. On frank-medium that means the DFlash2 drafter is not
-            running (check the unit's log for common_speculative_impl_draft_dflash),
-            and the endpoint is decoding autoregressively.
+            minutes. On the llama.cpp endpoints that means the drafter is not
+            running (check the unit's log for the
+            "adding speculative implementation" line — draft-mtp on both
+            endpoints as of 2026-09-17), and the endpoint is decoding
+            autoregressively.

      # The endpoint, not a replica: one process serves each port now, so a
      # restart of the unit is a scrape gap, not a failover.
@@ -21,7 +21,7 @@ load helpers

 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter)
+UNITS=(prometheus grafana loki alloy node-exporter alertmanager)

 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
    awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }

-@test "all five observability units exist under dankclaw/observability/quadlet/" {
+@test "all six observability units exist under dankclaw/observability/quadlet/" {
    local svc
    for svc in "${UNITS[@]}"; do
        [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -72,10 +72,11 @@ service_section() {
    ! grep -qE '^PublishPort=' "$UNIT_DIR/alloy.container" || true
 }

-@test "the three HTTP UIs publish on loopback only" {
+@test "the four HTTP UIs publish on loopback only" {
    grep -q '^PublishPort=127.0.0.1:9090:9090$' "$UNIT_DIR/prometheus.container"
    grep -q '^PublishPort=127.0.0.1:3002:3000$' "$UNIT_DIR/grafana.container"
    grep -q '^PublishPort=127.0.0.1:3100:3100$' "$UNIT_DIR/loki.container"
+    grep -q '^PublishPort=127.0.0.1:9093:9093$' "$UNIT_DIR/alertmanager.container"
 }

@test "the homepage status timer's probe targets are the published loopback ports" {
@@ -88,6 +89,7 @@ service_section() {
    grep -q '"probe": "127.0.0.1:3100"' "$registry"
    grep -q '"probe": "127.0.0.1:9100"' "$registry"
    grep -q '"probe": "127.0.0.1:12345"' "$registry"
+    grep -q '"probe": "127.0.0.1:9093"' "$registry"
 }

@test "node-exporter binds the host namespace and both of its consumers" {
@@ -142,7 +144,7 @@ service_section() {
    # unit HealthCmd would reach podman as CMD-SHELL and fail for want of
    # /bin/sh. Its health command ships in the image instead.
    local svc
-    for svc in prometheus grafana alloy node-exporter; do
+    for svc in prometheus grafana alloy node-exporter alertmanager; do
        grep -q '^HealthCmd=' "$UNIT_DIR/$svc.container" \
            || { echo "$svc has no HealthCmd"; return 1; }
        grep -q '^HealthRetries=' "$UNIT_DIR/$svc.container" \
@@ -248,6 +250,7 @@ grafana http://127.0.0.1:3002/api/health
 loki http://127.0.0.1:3100/ready
 alloy http://127.0.0.1:12345/-/healthy
 node-exporter http://127.0.0.1:9100/metrics
+alertmanager http://127.0.0.1:9093/-/healthy
 PAIRS
 }

@@ -416,3 +419,67 @@ assert any('ALERTS' in e for e in exprs), exprs
    grep -q 'local unit_dir="$TARGET/observability/quadlet"' "$handler"
    grep -q 'install_quadlet observability "$unit_dir" "$service"' "$handler"
 }
+
+@test "alertmanager delivers to the Signal relay through the podman host gateway" {
+    # The relay listens on the HOST's loopback. From the container that
+    # address is only reachable as host.containers.internal (netavark DNATs
+    # published ports only — the LAN IP and the gateway are refused), so the
+    # webhook URL and the relay's listen port are a cross-file contract: if
+    # either half changes alone, every alert is dropped with no error
+    # anywhere.
+    local am="$REPO_ROOT/shared/prometheus/alertmanager.yml"
+    local relay="$REPO_ROOT/scripts/signal-relay.py"
+    [ -f "$am" ] && [ -f "$relay" ]
+    grep -q 'host.containers.internal:18093/alertmanager' "$am"
+    # The relay must listen exactly where the webhook points.
+    grep -q '18093' "$relay"
+    grep -q '"/alertmanager"' "$relay"
+}
+
+@test "prometheus ships alerts to alertmanager on the shared network" {
+    # Container-name address, like grafana and loki: the published port is
+    # loopback-only and would be wrong from inside Prometheus.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'alertmanagers:' "$cfg"
+    grep -q '"alertmanager:9093"' "$cfg"
+}
+
+@test "the signal relay is a boot-persistent user unit the deployer installs" {
+    # Plain user unit, not a quadlet: it runs python3 from the checkout and
+    # can be enabled (generated quadlet units cannot on this host). Linger is
+    # on for dank, so [Install] + enable is the whole boot story.
+    local unit="$REPO_ROOT/scripts/signal-relay.service"
+    local handler="$REPO_ROOT/scripts/deploy-remote.sh"
+    [ -f "$unit" ]
+    grep -q '^WantedBy=default.target$' "$unit"
+    grep -q '^Restart=always$' "$unit"
+    grep -q '^ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/signal-relay.py$' "$unit"
+    grep -q '^install_signal_relay() {' "$handler"
+    grep -q 'systemctl --user enable signal-relay.service' "$handler"
+    grep -q 'systemctl --user restart signal-relay.service' "$handler"
+    # The alertmanager deploy is what reaches the relay on the host.
+    grep -q 'install_signal_relay' <(sed -n '/^deploy_observability()/,/^}/p' "$handler")
+}
+
+@test "the relay sends through the existing signal-cli account, loopback only" {
+    # The credential is the account state under ~/.local/share/signal-cli;
+    # the number and account name are numbers, not secrets. The listener must
+    # stay on loopback — an unauthenticated message-sending endpoint on a
+    # host interface is the failure this whole unit family exists to avoid.
+    local relay="$REPO_ROOT/scripts/signal-relay.py"
+    grep -q 'signal-cli' "$relay"
+    grep -q 'SIGNAL_CLI = "/usr/local/bin/signal-cli"' "$relay"
+    grep -q 'SIGNAL_ACCOUNT = "318311"' "$relay"
+    grep -q 'LISTEN_ADDR = ("127.0.0.1", 18093)' "$relay"
+}
+
+@test "the planner selects alertmanager for its config and relay files" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths shared/prometheus/alertmanager.yml
+    [ "$status" -eq 0 ]
+    echo "$output" | grep -qx 'alertmanager'
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths scripts/signal-relay.py
+    [ "$status" -eq 0 ]
+    [ "$output" = "alertmanager" ]
+}