feat(observability): make the five Dankclaw observability containers quadlets
prometheus, grafana, loki, alloy and node-exporter ran only from dankclaw/compose.yaml. Rootless podman does not restart bare containers, so they did not survive the 2026-09-11 reboot for the amd_iommu=off change and had to be restored by hand — the same gap that had already cost the fleet Caddy and Pi-hole, which is why the boot-persistence rule exists. Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/, installed and dispatched by deploy_observability in deploy-remote.sh, with the compose definitions retired so only one owner exists per container name. Images, mounts, arguments and container names are unchanged, so Caddy's proxies and Prometheus' own scrape config keep resolving. Two posture changes, both about the unauthenticated surface: * The three HTTP UIs published on 0.0.0.0 with no authentication of any kind (Prometheus even with the lifecycle endpoint enabled). They now publish on 127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy reaches them by name over dankclaw_default, and the host-side homepage status timer probes loopback. Nothing legitimately used the LAN port. * node-exporter bound :9100 on every interface; it now binds exactly its two consumers, loopback (homepage timer) and the LAN address (Prometheus, which runs in a container and cannot reach the host's loopback). alloy gains a reachable health endpoint: its UI listens on the container's own loopback and alloy 1.x refuses a config block for it, so the compose service had nothing to probe — the homepage status card for alloy read 'down' continuously. The unit moves the UI with a command-line flag and publishes 127.0.0.1:12345. grafana keeps its Authentik OAuth client secret without receiving the whole .env: systemd expands the value from a [Service] EnvironmentFile, which never reaches the container.
This commit is contained in:
Frank committed
1 parent
549bc57df2
commit
dab81b91a3
11 files changed
+655
-160
No files matched your search
@@ -0,0 +1,63 @@
|
||||
# Quadlet unit: alloy (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
|
||||
# Unit name: alloy.service (container name: alloy)
|
||||
#
|
||||
# Log shipper. This replaces the compose service of the same name in
|
||||
# dankclaw/compose.yaml ("Observability" block).
|
||||
#
|
||||
# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
|
||||
# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
|
||||
# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
|
||||
# boot from reaching for that local tag at a registry.
|
||||
#
|
||||
# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
|
||||
# resolves the container name loki:3100.
|
||||
#
|
||||
# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
|
||||
# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
|
||||
# refuses the ui block), so the compose service had no reachable health
|
||||
# endpoint at all: the homepage status card for alloy reported "down"
|
||||
# continuously and there was nothing for a deploy to wait on. The command
|
||||
# below therefore moves the UI to the container's wildcard address and the
|
||||
# port is published on host loopback only, which puts alloy in the same class
|
||||
# as the other three: reachable from the host, by Caddy over the private
|
||||
# network, and by nothing off-host. Alloy has no authentication, which is
|
||||
# exactly why it is not published wider.
|
||||
#
|
||||
# SOURCES. The host journald socket and the host log/container-storage trees
|
||||
# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
|
||||
# lives at /opt/alloy/config/config.alloy, installed from
|
||||
# shared/alloy/config.alloy on every deploy.
|
||||
#
|
||||
# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
|
||||
# client fails and its log pipeline backs up.
|
||||
|
||||
[Unit]
|
||||
Description=Alloy log shipper (Dankclaw)
|
||||
After=network-online.target loki.service
|
||||
Wants=network-online.target loki.service
|
||||
|
||||
[Container]
|
||||
ContainerName=alloy
|
||||
Image=localhost/franksdank/alloy:latest
|
||||
Pull=never
|
||||
Network=dankclaw_default
|
||||
Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
|
||||
PublishPort=127.0.0.1:12345:12345
|
||||
Volume=/opt/alloy/config:/etc/alloy
|
||||
Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
|
||||
Volume=/var/log:/var/log:ro
|
||||
Volume=/var/lib/containers:/var/lib/containers:ro
|
||||
HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,72 @@
|
||||
# Quadlet unit: grafana (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/grafana.container
|
||||
# Unit name: grafana.service (container name: grafana)
|
||||
#
|
||||
# Dashboards. This replaces the compose service of the same name in
|
||||
# dankclaw/compose.yaml ("Observability" block).
|
||||
#
|
||||
# PUBLISHING. Bound to 127.0.0.1 for the same reason as prometheus: the browser
|
||||
# path is https://grafana.franksdank.cloud (Caddy → Anubis → Authentik outpost →
|
||||
# grafana:3000 over dankclaw_default) and the host-side homepage status timer
|
||||
# probes 127.0.0.1:3002. Publishing on 0.0.0.0 as the compose service did left
|
||||
# an off-host path into Grafana that skips the guard chain entirely.
|
||||
#
|
||||
# AUTH. Grafana is an OAuth2 relying party against Authentik (OAuth2Provider
|
||||
# "grafana", confidential client). The guard chain fronts unauthenticated
|
||||
# traffic; the login itself happens through the Authentik button inside
|
||||
# Grafana. The client secret is NOT written here: it is expanded by systemd
|
||||
# from the root .env (EnvironmentFile in [Service], never in [Container], so
|
||||
# the container receives exactly this one secret and not the whole file).
|
||||
# Without that .env entry the unit will not start — the same dependency every
|
||||
# other rendered-secret deploy has.
|
||||
#
|
||||
# STATE. /opt/grafana/data holds grafana.db. /opt/grafana/provisioning is
|
||||
# mounted read-only and was EMPTY as of 2026-09-11, and grafana.db reported
|
||||
# zero datasources and zero dashboards: this Grafana is an empty install whose
|
||||
# only configured account is the local admin. Nothing here queries Prometheus
|
||||
# or Loki yet.
|
||||
|
||||
[Unit]
|
||||
Description=Grafana dashboards (Dankclaw)
|
||||
After=network-online.target prometheus.service
|
||||
Wants=network-online.target prometheus.service
|
||||
|
||||
[Container]
|
||||
ContainerName=grafana
|
||||
Image=docker.io/grafana/grafana:latest
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
PublishPort=127.0.0.1:3002:3000
|
||||
Environment=GF_SERVER_ROOT_URL=https://grafana.franksdank.cloud
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_ENABLED=true
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_NAME=Authentik
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana-21ca190cde75d60f
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
|
||||
# Global authorize path: this Authentik build resolves the provider by the
|
||||
# client_id query parameter — /application/o/<slug>/authorize/ does not exist.
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
|
||||
# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
|
||||
Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
|
||||
Volume=/opt/grafana/data:/var/lib/grafana
|
||||
Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
HealthCmd=wget --spider -q http://localhost:3000/api/health
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=60s
|
||||
|
||||
[Service]
|
||||
# systemd-side only: supplies ${GRAFANA_OAUTH_CLIENT_SECRET} for expansion
|
||||
# above. The container does not receive this file.
|
||||
EnvironmentFile=/home/dank/serverAppConfigs/.env
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,57 @@
|
||||
# Quadlet unit: loki (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/loki.container
|
||||
# Unit name: loki.service (container name: loki)
|
||||
#
|
||||
# Log store. This replaces the compose service of the same name in
|
||||
# dankclaw/compose.yaml ("Observability" block).
|
||||
#
|
||||
# IMAGE. franksdank/loki is built locally by deploy-remote.sh
|
||||
# (build_compose_image) from dankclaw/loki/Containerfile: upstream
|
||||
# grafana/loki plus a static busybox, because loki is distroless and the
|
||||
# healthcheck has nothing to run inside it. Pull=never keeps a boot from
|
||||
# trying to fetch that local tag from a registry. The localhost/ prefix is
|
||||
# required — this host's quadlet path cannot resolve a registry-prefixed ref
|
||||
# from local storage at create time.
|
||||
#
|
||||
# PUBLISHING. Bound to 127.0.0.1: Alloy pushes to loki:3100 over
|
||||
# dankclaw_default, Caddy proxies the guarded vhost to loki:3100 the same way,
|
||||
# and the homepage status timer probes 127.0.0.1:3100. Loki has
|
||||
# auth_enabled: false, so the compose service's 0.0.0.0 binding was an
|
||||
# unauthenticated log-ingest and read API on every host interface.
|
||||
#
|
||||
# CONFIG. /opt/loki/config/loki-config.yaml is installed from
|
||||
# shared/loki/loki-config.yaml on every deploy (single-binary, filesystem
|
||||
# storage, 30-day retention). Chunks and rules live in /opt/loki/data.
|
||||
#
|
||||
# Rootless: container root maps to the invoking user (dank), who owns
|
||||
# /opt/loki/data; the image's default user (10001) maps to a subuid that
|
||||
# cannot write it.
|
||||
|
||||
[Unit]
|
||||
Description=Loki log store (Dankclaw)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=loki
|
||||
Image=localhost/franksdank/loki:latest
|
||||
Pull=never
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
Exec=-config.file=/etc/loki/loki-config.yaml
|
||||
PublishPort=127.0.0.1:3100:3100
|
||||
Volume=/opt/loki/config:/etc/loki
|
||||
Volume=/opt/loki/data:/loki
|
||||
HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:3100/ready
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,52 @@
|
||||
# Quadlet unit: node-exporter (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/node-exporter.container
|
||||
# Unit name: node-exporter.service (container name: node-exporter)
|
||||
#
|
||||
# Host metrics source for Prometheus. This replaces the compose service of the
|
||||
# same name in dankclaw/compose.yaml ("Observability" block).
|
||||
#
|
||||
# HOST NAMESPACE. It must run in the host network namespace to report the
|
||||
# host's own interfaces/connections, and --pid=host for the processes and
|
||||
# cgroups collectors. "localhost" inside the container is therefore the host's
|
||||
# loopback, which is what the homepage status timer probes.
|
||||
#
|
||||
# LISTEN ADDRESSES. Until 2026-09-11 this bound :9100 on every interface. It
|
||||
# now binds exactly two: loopback for the homepage status probe, and the LAN
|
||||
# address because Prometheus — which runs in a container on dankclaw_default —
|
||||
# scrapes http://192.168.0.143:9100 (shared/prometheus/prometheus.yml,
|
||||
# job "node-dankclaw"). node_exporter itself has no authentication, so keeping
|
||||
# the wildcard bind off the table matters: this is the second half of the
|
||||
# pairing that removes the unauthenticated observability surface, and both
|
||||
# consumers are accounted for. Dankserver runs its own copy of this container
|
||||
# (scraped by the "node-dankserver" job) from its own compose stack.
|
||||
#
|
||||
# MOUNTS. The container reads the host's /proc, /sys and / as /host/* read-only
|
||||
# and is pointed at them with --path.*; it writes nothing anywhere.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus node exporter (Dankclaw host metrics)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=node-exporter
|
||||
Image=docker.io/prom/node-exporter:latest
|
||||
Network=host
|
||||
PodmanArgs=--pid=host
|
||||
Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
|
||||
Volume=/proc:/host/proc:ro
|
||||
Volume=/sys:/host/sys:ro
|
||||
Volume=/:/rootfs:ro
|
||||
HealthCmd=wget --spider -q http://localhost:9100/metrics
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,56 @@
|
||||
# Quadlet unit: prometheus (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/prometheus.container
|
||||
# Unit name: prometheus.service (container name: prometheus)
|
||||
#
|
||||
# The homelab metrics store. This replaces the compose service of the same
|
||||
# name in dankclaw/compose.yaml ("Observability" block): these five containers
|
||||
# were started by hand with podman-compose and never came back after a reboot,
|
||||
# which the 2026-09-11 reboot for the amd_iommu=off change made obvious.
|
||||
#
|
||||
# PUBLISHING. The published port is bound to 127.0.0.1 deliberately. Prometheus
|
||||
# is reached by Caddy over the dankclaw_default network (the Caddyfile's
|
||||
# post-auth handler proxies to prometheus:9090) and by the host-side homepage
|
||||
# status timer over 127.0.0.1. The compose service published on 0.0.0.0, which
|
||||
# put an UNAUTHENTICATED metrics API — with the lifecycle endpoint enabled —
|
||||
# on every host interface: a bypass around the Anubis + Authentik guard on
|
||||
# prometheus.franksdank.cloud. No off-host consumer needs it.
|
||||
#
|
||||
# CONFIG. /opt/prometheus/config/prometheus.yml is installed from
|
||||
# shared/prometheus/prometheus.yml by deploy-remote.sh on every deploy, so the
|
||||
# repo copy is the source of truth. STATE (30-day retention) lives in
|
||||
# /opt/prometheus/data.
|
||||
#
|
||||
# Rootless: container root maps to the invoking user (dank) in this user
|
||||
# namespace and dank owns /opt/prometheus/data. The image's default user
|
||||
# (nobody) maps to a subuid that cannot write it.
|
||||
#
|
||||
# There are no alerting rules and no Alertmanager anywhere in this stack: the
|
||||
# scrape config is the whole of its configuration.
|
||||
|
||||
[Unit]
|
||||
Description=Prometheus metrics store (Dankclaw)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=prometheus
|
||||
Image=docker.io/prom/prometheus:latest
|
||||
Network=dankclaw_default
|
||||
User=0:0
|
||||
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
|
||||
PublishPort=127.0.0.1:9090:9090
|
||||
Volume=/opt/prometheus/config:/etc/prometheus
|
||||
Volume=/opt/prometheus/data:/prometheus
|
||||
HealthCmd=wget --spider -q http://localhost:9090/-/healthy
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=30s
|
||||
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
Reference in new issue
Block a user