Files

T

Frank 5c1006dda6 fix(monitoring): scrape node-dankclaw via host.containers.internal

node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.

2026-09-21 11:09:03 -07:00

4.4 KiB

Raw Blame History

Prometheus

Prometheus collects metrics from every service on Dankclaw by scraping their /metrics endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy
Data	`/opt/prometheus/data` (TSDB)
Retention	`--storage.tsdb.retention.time=30d`
Alerting	5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`	`host.containers.internal:9100`	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
`node-dankserver`	`192.168.0.235:9100`	host metrics from Node-Exporter — a different machine, so the LAN address works
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-medium`	`192.168.0.235:7770`	the LLM fleet — `vllm:` series; a different machine, so the LAN address works
`frank-large`, `frank-small`	`host.containers.internal:7770`, `:7771`	the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway

Three jobs read down, and they are not outages (audited 2026-09-16):

Job	Target	Why it never reports up
`comfyui`	`comfyui:8188`	scrapes `/`, which is HTML — "unsupported Content-Type"
`services`	`192.168.0.143:8081`, `:3005`	same: up/down intent, implemented as a scrape
`gitea`	`192.168.0.143:3000/metrics`	Gitea's exporter is off until `[metrics] ENABLED = true`

A scrape cannot express "is this HTTP service answering?" — that is what a probe is for, and the remedy is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives the guarded vhosts a real uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config`
`/prometheus`	`/opt/prometheus/data`

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on.

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes 192.168.0.143:9100, which is exactly the LAN address node-exporter.container binds (Network=host, --web.listen-address twice). Change one without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats seam pins the pairing.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here).
The lifecycle endpoint is enabled (--web.enable-lifecycle), which is why this API must never be published beyond loopback.

4.4 KiB Raw Blame History