# Quadlet unit: prometheus (rootless, boot-persistent). # # Install path on dankclaw: ~/.config/containers/systemd/prometheus.container # Unit name: prometheus.service (container name: prometheus) # # The homelab metrics store. This replaces the compose service of the same # name in dankclaw/compose.yaml ("Observability" block): these five containers # were started by hand with podman-compose and never came back after a reboot, # which the 2026-09-11 reboot for the amd_iommu=off change made obvious. # # PUBLISHING. The published port is bound to 127.0.0.1 deliberately. Prometheus # is reached by Caddy over the dankclaw_default network (the Caddyfile's # post-auth handler proxies to prometheus:9090) and by the host-side homepage # status timer over 127.0.0.1. The compose service published on 0.0.0.0, which # put an UNAUTHENTICATED metrics API — with the lifecycle endpoint enabled — # on every host interface: a bypass around the Anubis + Authentik guard on # prometheus.franksdank.cloud. No off-host consumer needs it. # # CONFIG. /opt/prometheus/config/prometheus.yml is installed from # shared/prometheus/prometheus.yml by deploy-remote.sh on every deploy, so the # repo copy is the source of truth. STATE (30-day retention) lives in # /opt/prometheus/data. # # Rootless: container root maps to the invoking user (dank) in this user # namespace and dank owns /opt/prometheus/data. The image's default user # (nobody) maps to a subuid that cannot write it. # # RULES. Five LLM alert rules and one recording rule load from # /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the # deployer). Delivery went live 2026-09-17: the alerting block in # prometheus.yml points at alertmanager:9093, which webhooks every firing # group to the host-side Signal relay (192.168.0.143:18093, reached from # the container as host.containers.internal), which sends through signal-cli account 318311. # The Grafana dashboard's ALERTS panel remains the second surface. # # The config volume is also why rules belong in git: editing them in the UI is # lost on the next deploy, since the deployer overwrites the whole directory. [Unit] Description=Prometheus metrics store (Dankclaw) After=network-online.target Wants=network-online.target [Container] ContainerName=prometheus Image=docker.io/prom/prometheus:latest Network=dankclaw_default User=0:0 Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle PublishPort=127.0.0.1:9090:9090 Volume=/opt/prometheus/config:/etc/prometheus Volume=/opt/prometheus/data:/prometheus HealthCmd=wget --spider -q http://localhost:9090/-/healthy HealthInterval=30s HealthTimeout=10s HealthRetries=3 HealthStartPeriod=30s [Service] Restart=always RestartSec=10 [Install] WantedBy=default.target