Frank
139a3cd4fa
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
71 lines
3.1 KiB
INI
71 lines
3.1 KiB
INI
# Quadlet unit: prometheus (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/prometheus.container
|
|
# Unit name: prometheus.service (container name: prometheus)
|
|
#
|
|
# The homelab metrics store. This replaces the compose service of the same
|
|
# name in dankclaw/compose.yaml ("Observability" block): these five containers
|
|
# were started by hand with podman-compose and never came back after a reboot,
|
|
# which the 2026-09-11 reboot for the amd_iommu=off change made obvious.
|
|
#
|
|
# PUBLISHING. The published port is bound to 127.0.0.1 deliberately. Prometheus
|
|
# is reached by Caddy over the dankclaw_default network (the Caddyfile's
|
|
# post-auth handler proxies to prometheus:9090) and by the host-side homepage
|
|
# status timer over 127.0.0.1. The compose service published on 0.0.0.0, which
|
|
# put an UNAUTHENTICATED metrics API — with the lifecycle endpoint enabled —
|
|
# on every host interface: a bypass around the Anubis + Authentik guard on
|
|
# prometheus.franksdank.cloud. No off-host consumer needs it.
|
|
#
|
|
# CONFIG. /opt/prometheus/config/prometheus.yml is installed from
|
|
# shared/prometheus/prometheus.yml by deploy-remote.sh on every deploy, so the
|
|
# repo copy is the source of truth. STATE (30-day retention) lives in
|
|
# /opt/prometheus/data.
|
|
#
|
|
# Rootless: container root maps to the invoking user (dank) in this user
|
|
# namespace and dank owns /opt/prometheus/data. The image's default user
|
|
# (nobody) maps to a subuid that cannot write it.
|
|
#
|
|
# RULES. Five LLM alert rules and one recording rule load from
|
|
# /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
|
|
# deployer). Delivery went live 2026-09-17: the alerting block in
|
|
# prometheus.yml points at alertmanager:9093, which webhooks every firing
|
|
# group to the host-side Signal relay (192.168.0.143:18093, reached from
|
|
# the container as host.containers.internal), which sends through signal-cli account 318311.
|
|
# The Grafana dashboard's ALERTS panel remains the second surface.
|
|
#
|
|
# The config volume is also why rules belong in git: editing them in the UI is
|
|
# lost on the next deploy, since the deployer overwrites the whole directory.
|
|
|
|
[Unit]
|
|
Description=Prometheus metrics store (Dankclaw)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=prometheus
|
|
Image=docker.io/prom/prometheus:latest
|
|
Network=dankclaw_default
|
|
# Name the podman gateway so scrape jobs can reach same-host targets (the
|
|
# netavark hairpin gap bars this container from dialing its own host's LAN
|
|
# IP; the cadvisor/gpu jobs and the blackbox scrape all ride this name).
|
|
# Same mechanism the hermes Caddy upstream learned the hard way (Container
|
|
# Host Reachability in the vault).
|
|
HostAddr=host.containers.internal=169.254.1.2
|
|
User=0:0
|
|
Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
|
|
PublishPort=127.0.0.1:9090:9090
|
|
Volume=/opt/prometheus/config:/etc/prometheus
|
|
Volume=/opt/prometheus/data:/prometheus
|
|
HealthCmd=wget --spider -q http://localhost:9090/-/healthy
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=30s
|
|
|
|
[Service]
|
|
Restart=always
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|