feat(observability): make the five Dankclaw observability containers quadlets

prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.

Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.

Two posture changes, both about the unauthenticated surface:

* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
  (Prometheus even with the lifecycle endpoint enabled). They now publish on
  127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
  reaches them by name over dankclaw_default, and the host-side homepage
  status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
  consumers, loopback (homepage timer) and the LAN address (Prometheus, which
  runs in a container and cannot reach the host's loopback).

alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.

grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
This commit is contained in:
Frank
2026-09-11 15:56:39 -07:00
parent 549bc57df2
commit dab81b91a3
11 changed files with 655 additions and 160 deletions
+10
View File
@@ -300,6 +300,16 @@ def services_for_path(path: str, target: str) -> set[str]:
# its unit and fleet settings live in one .container file.
_add(services, "frank-large")
return services
if len(parts) >= 2 and parts[1] == "observability":
# The five observability containers became Quadlet units on
# 2026-09-11. A unit change selects exactly its own service;
# SERVICE_ORDER already puts each dependency ahead of its
# consumer (prometheus before grafana, loki before alloy).
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3].removesuffix(".container")
if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter"}:
_add(services, name)
return services
if len(parts) >= 2 and parts[1] == "gitea":
if len(parts) >= 4 and parts[2] == "quadlet":
name = parts[3]