relay: bind 169.254.1.2 (host.containers.internal), not loopback

A socket bound to 127.0.0.1 refuses container traffic that arrives via the podman bridge: the first live webhook failed with 'dial tcp 169.254.1.2:18093: connection refused'. The relay now binds the host-gateway endpoint — link-local to the podman bridge, off the LAN — and the deployer's health probe follows it there.
2026-09-17 13:26:24 -07:00
parent 5f55efcb9d
commit 8f747a2104
9 changed files with 31 additions and 21 deletions
@@ -128,8 +128,8 @@ provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
 llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
 delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
 alertmanager.container`, loopback :9093) → host-side Signal relay
-(`scripts/signal-relay.py`, user unit, loopback :18093, reached from the
-container as `host.containers.internal`) → signal-cli account 318311. Five
+(`scripts/signal-relay.py`, user unit, 169.254.1.2:18093 — the
+host.containers.internal endpoint) → signal-cli account 318311. Five
 Prometheus targets read down for structural reasons (HTML-scraped "probes",
 Gitea metrics off); the remedy is a blackbox exporter. See
 `serverAppConfigs-docs/Infrastructure/Observability.md`.
@@ -29,8 +29,8 @@
 # /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
 # deployer). Delivery went live 2026-09-17: the alerting block in
 # prometheus.yml points at alertmanager:9093, which webhooks every firing
-# group to the host-side Signal relay (127.0.0.1:18093, reached as
-# host.containers.internal), which sends through signal-cli account 318311.
+# group to the host-side Signal relay (169.254.1.2:18093, the
+# host.containers.internal endpoint), which sends through signal-cli account 318311.
 # The Grafana dashboard's ALERTS panel remains the second surface.
 #
 # The config volume is also why rules belong in git: editing them in the UI is
@@ -232,15 +232,16 @@ install_signal_relay() {
    systemctl --user enable signal-relay.service
    systemctl --user restart signal-relay.service
    # The interpreter needs a moment to bind after restart; a one-shot curl
-    # raced the socket on 2026-09-17 and failed a healthy deploy.
+    # raced the socket on 2026-09-17 and failed a healthy deploy. The relay
+    # binds 169.254.1.2 (host.containers.internal), not loopback.
    local i
    for i in $(seq 1 15); do
-        if curl -fsS -m 5 http://127.0.0.1:18093/health >/dev/null; then
+        if curl -fsS -m 5 http://169.254.1.2:18093/health >/dev/null; then
            return 0
        fi
        sleep 1
    done
-    echo "signal-relay is not answering on 127.0.0.1:18093" >&2
+    echo "signal-relay is not answering on 169.254.1.2:18093" >&2
    return 1
 }

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 """Alertmanager-to-Signal relay for the homelab (Dankclaw).
-
-Listens on 127.0.0.1:18093, receives Alertmanager's webhook payload, formats
-each alert group into a short message and hands it to signal-cli under the
+Listens on 169.254.1.2:18093 — the host side of host.containers.internal,
+the only address a bridge container can use to reach a host-local process —
+receives Alertmanager's webhook payload, formats
 existing account on this host. Runs as a systemd user unit
 (scripts/signal-relay.service); logs go to the journal.

@@ -20,7 +20,13 @@ import sys
 import time
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

-LISTEN_ADDR = ("127.0.0.1", 18093)
+# 169.254.1.2 is the host endpoint of host.containers.internal: a socket
+# bound to 127.0.0.1 refuses container traffic that arrives via the podman
+# bridge (measured 2026-09-17: "dial tcp 169.254.1.2:18093: connection
+# refused"). The address is link-local to the podman bridge — not on any
+# LAN interface, not on loopback, reachable only from this host's
+# containers.
+LISTEN_ADDR = ("169.254.1.2", 18093)
 SIGNAL_CLI = "/usr/local/bin/signal-cli"
 SIGNAL_ACCOUNT = "318311"
 SIGNAL_TO = "+12062476336"
@@ -19,7 +19,7 @@ Alertmanager receives firing rules from Prometheus, groups and dedupes them, and

 ## Why the webhook address is `host.containers.internal`

-The Signal relay is a host-side process on the host's **loopback** (127.0.0.1:18093). From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — so the LAN IP and the podman gateway both refuse an unpublished host bind. `host.containers.internal` (169.254.1.2) is the one address that reaches a host-local listener from a bridge container. The relay listens on loopback only, so the unauthenticated webhook endpoint stays off every host interface.
+The Signal relay is a host-side process, and the address it must bind is the one detail this design gets wrong once: **127.0.0.1 does not work**. From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — the LAN IP and the podman gateway are both refused for an unpublished host bind. `host.containers.internal` (169.254.1.2) is the one address that reaches a host-local listener from a bridge container, so the relay binds *that* address (measured 2026-09-17: a loopback bind refused the first webhook with `dial tcp 169.254.1.2:18093: connection refused`). 169.254.1.2 is link-local to the podman bridge — not on loopback, not on any LAN interface, reachable only from this host's containers — so the unauthenticated webhook endpoint stays off the wire.

 ## The Signal relay

@@ -36,7 +36,7 @@ The Signal relay is a host-side process on the host's **loopback** (127.0.0.1:18
 ./scripts/deploy.sh dankclaw --service alertmanager
 ```

-`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, enables and restarts it, then waits on `127.0.0.1:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.
+`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, enables and restarts it, then waits on `169.254.1.2:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.

 ## Gotchas

@@ -40,7 +40,7 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in

 - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the netavark host-gateway address, because the relay listens on the host's loopback and bridge containers cannot reach a host bind at the LAN IP. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
+- **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds 169.254.1.2, the host-gateway endpoint: a loopback bind refuses container traffic (measured on the first deploy), and a host-interface bind would put an unauthenticated endpoint on the LAN. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
@@ -2,7 +2,8 @@
 # Deployed to /opt/alertmanager/config/alertmanager.yml on Dankclaw.
 #
 # The one delivery path is Signal. The webhook lands on the host-side
-# signal-relay (a systemd user unit listening on 127.0.0.1:18093), which
+# signal-relay (a systemd user unit listening on 169.254.1.2:18093, the
+# host.containers.internal endpoint), which
 # formats the group and hands it to signal-cli under Frank's existing
 # account. There is deliberately no second receiver: a duplicate channel is
 # how an alert starts arriving twice and getting muted everywhere.
@@ -14,8 +14,8 @@ rule_files:

 # The delivery half. Alertmanager runs on the same dankclaw_default network,
 # so it is addressed by container name, like grafana and loki below. Its
-# single receiver webhooks the host-side signal-relay (loopback :18093,
-# reachable from the container only as host.containers.internal).
+# single receiver webhooks the host-side signal-relay (169.254.1.2:18093,
+# the host.containers.internal endpoint).
 alerting:
  alertmanagers:
    - static_configs:
@@ -461,16 +461,18 @@ assert any('ALERTS' in e for e in exprs), exprs
    grep -q 'install_signal_relay' <(sed -n '/^deploy_observability()/,/^}/p' "$handler")
 }

-@test "the relay sends through the existing signal-cli account, loopback only" {
+@test "the relay sends through the existing signal-cli account, podman-bridge only" {
    # The credential is the account state under ~/.local/share/signal-cli;
    # the number and account name are numbers, not secrets. The listener must
-    # stay on loopback — an unauthenticated message-sending endpoint on a
-    # host interface is the failure this whole unit family exists to avoid.
+    # bind 169.254.1.2 (host.containers.internal): a loopback bind refuses
+    # container traffic (measured 2026-09-17), and a host-interface bind
+    # would put an unauthenticated message-sending endpoint on the LAN —
+    # the failure this whole unit family exists to avoid.
    local relay="$REPO_ROOT/scripts/signal-relay.py"
    grep -q 'signal-cli' "$relay"
    grep -q 'SIGNAL_CLI = "/usr/local/bin/signal-cli"' "$relay"
    grep -q 'SIGNAL_ACCOUNT = "318311"' "$relay"
-    grep -q 'LISTEN_ADDR = ("127.0.0.1", 18093)' "$relay"
+    grep -q 'LISTEN_ADDR = ("169.254.1.2", 18093)' "$relay"
 }

@test "the planner selects alertmanager for its config and relay files" {