feat(observability): make the five Dankclaw observability containers quadlets

prometheus, grafana, loki, alloy and node-exporter ran only from dankclaw/compose.yaml. Rootless podman does not restart bare containers, so they did not survive the 2026-09-11 reboot for the amd_iommu=off change and had to be restored by hand — the same gap that had already cost the fleet Caddy and Pi-hole, which is why the boot-persistence rule exists. Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/, installed and dispatched by deploy_observability in deploy-remote.sh, with the compose definitions retired so only one owner exists per container name. Images, mounts, arguments and container names are unchanged, so Caddy's proxies and Prometheus' own scrape config keep resolving. Two posture changes, both about the unauthenticated surface: * The three HTTP UIs published on 0.0.0.0 with no authentication of any kind (Prometheus even with the lifecycle endpoint enabled). They now publish on 127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy reaches them by name over dankclaw_default, and the host-side homepage status timer probes loopback. Nothing legitimately used the LAN port. * node-exporter bound :9100 on every interface; it now binds exactly its two consumers, loopback (homepage timer) and the LAN address (Prometheus, which runs in a container and cannot reach the host's loopback). alloy gains a reachable health endpoint: its UI listens on the container's own loopback and alloy 1.x refuses a config block for it, so the compose service had nothing to probe — the homepage status card for alloy read 'down' continuously. The unit moves the UI with a command-line flag and publishes 127.0.0.1:12345. grafana keeps its Authentik OAuth client secret without receiving the whole .env: systemd expands the value from a [Service] EnvironmentFile, which never reaches the container.
2026-09-11 15:56:39 -07:00
1 parent 549bc57df2
commit dab81b91a3
@@ -143,134 +143,18 @@ services:
  # not compose services. See gitea/quadlet/ and ../docs/Dankclaw/Gitea.md.

  # --- Observability -------------------------------------------------------
-  prometheus:
-    image: docker.io/prom/prometheus:latest
-    container_name: prometheus
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/prometheus/data. The image's default user (nobody) maps to
-    # a subuid that cannot write it.
-    user: "0:0"
-    command:
-      - --config.file=/etc/prometheus/prometheus.yml
-      - --storage.tsdb.path=/prometheus
-      - --storage.tsdb.retention.time=30d
-      - --web.enable-lifecycle
-    ports:
-      - "9090:9090"
-    volumes:
-      - /opt/prometheus/config:/etc/prometheus
-      - /opt/prometheus/data:/prometheus
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  grafana:
-    image: docker.io/grafana/grafana:latest
-    container_name: grafana
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/grafana/data. The image's default user (472) maps to a
-    # subuid that cannot write it.
-    user: "0:0"
-    environment:
-      GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud"
-      # Auth model = OAuth2 relying party against authentik (OAuth2Provider
-      # name "grafana", pk 9, confidential client_type, created via ak shell).
-      # Chosen over auth_proxy because the embedded outpost carries no provider
-      # token (token=None), so no identity header ever reaches Grafana — RP mode
-      # works regardless of outpost topology. The public vhost stays behind the
-      # Anubis challenge + outpost gate; login happens inside Grafana via the
-      # Authentik button. Users auto-upsert as Viewer (no role mapping yet);
-      # the local admin account remains the only Admin.
-      GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
-      GF_AUTH_GENERIC_OAUTH_NAME: "Authentik"
-      GF_AUTH_GENERIC_OAUTH_CLIENT_ID: "grafana-21ca190cde75d60f"
-      GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: "${GRAFANA_OAUTH_CLIENT_SECRET}"
-      # Global authorize path: this fork resolves the provider by client_id
-      # query param — /application/o/<slug>/authorize/ does not exist here.
-      GF_AUTH_GENERIC_OAUTH_AUTH_URL: "https://auth.franksdank.cloud/application/o/authorize/"
-      GF_AUTH_GENERIC_OAUTH_TOKEN_URL: "https://auth.franksdank.cloud/application/o/token/"
-      # ?api=1 = introspection-style userinfo: returns all custom claims
-      # (entitlements/groups) for access tokens, not just the standard set.
-      GF_AUTH_GENERIC_OAUTH_API_URL: "https://auth.franksdank.cloud/application/o/userinfo/?api=1"
-      GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: "false"
-      GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT: "true"
-      GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG: "true"
-      GF_AUTH_GENERIC_OAUTH_ORG_ID: "1"
-    ports:
-      - "3002:3000"
-    volumes:
-      - /opt/grafana/data:/var/lib/grafana
-      - /opt/grafana/provisioning:/etc/grafana/provisioning:ro
-    depends_on:
-      - prometheus
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  loki:
-    # Local build: grafana/loki + static busybox (healthcheck).
-    image: franksdank/loki:latest
-    container_name: loki
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/loki/data. The image's default user (10001) maps to a
-    # subuid that cannot write it.
-    user: "0:0"
-    command: ["-config.file=/etc/loki/loki-config.yaml"]
-    ports:
-      - "3100:3100"
-    volumes:
-      - /opt/loki/config:/etc/loki
-      - /opt/loki/data:/loki
-    healthcheck:
-      test: ["CMD", "/usr/local/bin/busybox", "wget", "--spider", "-q", "http://localhost:3100/ready"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  alloy:
-    # Local build: grafana/alloy + static busybox (healthcheck).
-    image: franksdank/alloy:latest
-    container_name: alloy
-    restart: unless-stopped
-    volumes:
-      - /opt/alloy/config:/etc/alloy
-      - /run/systemd/journal/socket:/run/systemd/journal/socket:ro
-      - /var/log:/var/log:ro
-      - /var/lib/containers:/var/lib/containers:ro
-    depends_on:
-      - loki
-    healthcheck:
-      test: ["CMD", "/usr/local/bin/busybox", "wget", "--spider", "-q", "http://localhost:12345/-/healthy"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  node-exporter:
-    image: docker.io/prom/node-exporter:latest
-    container_name: node-exporter
-    restart: unless-stopped
-    network_mode: host
-    pid: host
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-    command:
-      - --path.procfs=/host/proc
-      - --path.sysfs=/host/sys
-      - --path.rootfs=/rootfs
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9100/metrics"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
+  # Now boot-persistent Quadlet units, not compose services: prometheus,
+  # grafana, loki, alloy and node-exporter install from
+  # dankclaw/observability/quadlet/ and are dispatched by deploy_observability
+  # in scripts/deploy-remote.sh. They left compose on 2026-09-11, after a host
+  # reboot proved that compose-only containers do not come back on their own —
+  # the same gap that had already cost the fleet Caddy and Pi-hole.
+  #
+  # The three HTTP UIs publish on 127.0.0.1 only. Off-host access is through
+  # the guarded franksdank.cloud vhosts (Caddy reaches these containers by
+  # name over this network) and the host's own homepage status timer probes
+  # them over loopback. See
+  # serverAppConfigs-docs/Infrastructure/Observability.md.

  # --- Authentik (SSO / OIDC provider) -------------------------------------
  # Now boot-persistent Quadlet units, not compose services: authentik-