dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): make the five Dankclaw observability containers quadlets

Browse Source 
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.

Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.

Two posture changes, both about the unauthenticated surface:

* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
  (Prometheus even with the lifecycle endpoint enabled). They now publish on
  127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
  reaches them by name over dankclaw_default, and the host-side homepage
  status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
  consumers, loopback (homepage timer) and the LAN address (Prometheus, which
  runs in a container and cannot reach the host's loopback).

alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.

grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
This commit is contained in:
Frank
2026-09-11 15:56:39 -07:00
parent 549bc57df2
commit dab81b91a3
11 changed files with 655 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/caddyfile.bats
+30 -21
View File
@@ -192,18 +192,13 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
@test "observability containers run as container root (rootless = host user)" {
# In rootless podman, container root maps to the invoking user (dank),
# which owns the /opt data dirs. The images' default users (loki 10001,
# who owns the /opt data dirs. The images' default users (loki 10001,
# prometheus nobody, grafana 472) map to subuids that own nothing under
# /opt and cannot write the data dirs.
local compose="$REPO_ROOT/dankclaw/compose.yaml"
local svc block
local svc
for svc in prometheus loki grafana; do
block=$(awk -v s="^ $svc:\$" '
$0 ~ s { inblk = 1; next }
inblk && /^ [a-z]/ { inblk = 0 }
inblk' "$compose")
echo "$block" | grep -q 'user: "0:0"' || {
echo "service $svc does not run as user 0:0"; return 1;
grep -q '^User=0:0$' "$REPO_ROOT/dankclaw/observability/quadlet/$svc.container" || {
echo "$svc does not run as user 0:0"; return 1;
}
done
}
@@ -211,27 +206,41 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
# --- distroless healthchecks (loki/alloy carry a static busybox) ---
@test "loki and alloy run from local images carrying a static busybox" {
local compose="$REPO_ROOT/dankclaw/compose.yaml"
grep -q 'image: franksdank/loki:latest' "$compose" || { echo "loki image not local"; return 1; }
grep -q 'image: franksdank/alloy:latest' "$compose" || { echo "alloy image not local"; return 1; }
local img
local img unit cf
for img in loki alloy; do
local cf="$REPO_ROOT/dankclaw/$img/Containerfile"
unit="$REPO_ROOT/dankclaw/observability/quadlet/$img.container"
grep -q "^Image=localhost/franksdank/$img:latest$" "$unit" \
|| { echo "$img does not run the local image"; return 1; }
# The tag exists only in this host's storage: a boot must not reach
# for it at a registry.
grep -q '^Pull=never$' "$unit" \
|| { echo "$img would try to pull its local tag"; return 1; }
cf="$REPO_ROOT/dankclaw/$img/Containerfile"
[ -f "$cf" ] || { echo "missing $cf"; return 1; }
grep -q 'busybox' "$cf" || { echo "$cf does not install busybox"; return 1; }
done
}
@test "loki and alloy healthchecks use the busybox wget" {
local compose="$REPO_ROOT/dankclaw/compose.yaml"
local n
n=$(grep -c '/usr/local/bin/busybox' "$compose")
[ "$n" -ge 2 ] || { echo "expected >=2 busybox healthchecks, found $n"; return 1; }
# Both images are distroless-ish, so the healthcheck only works because
# the local Containerfile drops a static busybox into the image.
local n=0 unit
for unit in "$REPO_ROOT"/dankclaw/observability/quadlet/loki.container \
"$REPO_ROOT"/dankclaw/observability/quadlet/alloy.container; do
grep -q '^HealthCmd=/usr/local/bin/busybox wget' "$unit" && n=$((n + 1))
done
[ "$n" -eq 2 ] || { echo "expected 2 busybox healthchecks, found $n"; return 1; }
}
@test "alloy config declares no ui/server block (default UI listens on 127.0.0.1:12345)" {
@test "alloy's UI address comes from the unit, not the config file" {
# alloy 1.x refuses to configure the ui service from a config block
# ("service ui does not support being configured"). The default UI
# already listens where the in-container healthcheck probes.
# ("service ui does not support being configured"), so the unit moves the
# UI off the container's loopback with a command-line flag. Without that
# flag the port cannot be published and the service has no reachable
# health endpoint at all — which is why the homepage status card for
# alloy read "down" while it ran under compose.
! grep -qE '^(ui|server) \{' "$REPO_ROOT/shared/alloy/config.alloy"
local unit="$REPO_ROOT/dankclaw/observability/quadlet/alloy.container"
grep -q -- '--server.http.listen-addr=0.0.0.0:12345' "$unit"
grep -q '^PublishPort=127.0.0.1:12345:12345$' "$unit"
}