Files

T

Frank 701ab14f60 docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.

2026-09-11 16:10:27 -07:00

3.2 KiB

Raw Blame History

Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their /metrics endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy
Data	`/opt/prometheus/data` (TSDB)
Retention	`--storage.tsdb.retention.time=30d`
Alerting	none — no rule files, no Alertmanager anywhere in the stack

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`, `node-dankserver`	`192.168.0.143:9100`, `192.168.0.235:9100`	host metrics from each machine's Node-Exporter
`gitea`	`192.168.0.143:3000`	Gitea's built-in exporter, reached on its published LAN port
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network
`comfyui`	`comfyui:8188`	UI probe; ComfyUI has no native `/metrics`
`services`	`192.168.0.143:8081`, `:3005`	up/down probes for SearXNG and OpenWebUI

Targets are host:port, never URLs. For richer HTTP assertions, add the Prometheus blackbox exporter in probe mode rather than scraping HTML.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config`
`/prometheus`	`/opt/prometheus/data`

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on.

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes 192.168.0.143:9100, which is exactly the LAN address node-exporter.container binds (Network=host, --web.listen-address twice). Change one without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats seam pins the pairing.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here).
The lifecycle endpoint is enabled (--web.enable-lifecycle), which is why this API must never be published beyond loopback.

3.2 KiB Raw Blame History