serverAppConfigs/serverAppConfigs-docs/Infrastructure/Prometheus.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural.	infrastructure prometheus	2026-09-21

Prometheus

Prometheus collects metrics from the services listed below by scraping their /metrics endpoints, and it is the datasource behind the provisioned llm-fleet dashboard in Grafana (datasource uid prometheus, shared/grafana/provisioning/datasources/prometheus.yml:14-21). node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	docker.io/prom/prometheus:latest
Published	127.0.0.1:9090 → 9090, loopback only — never a wildcard (dankclaw/observability/quadlet/prometheus.container:50; ss -ltnp 2026-10-18 shows 127.0.0.1:9090 and no other bind; tests/observability_quadlet.bats:63 pins it)
Unit	dankclaw/observability/quadlet/prometheus.container → prometheus.service; Restart=always/RestartSec=10 + WantedBy=default.target (prometheus.container:59-64), installed at ~/.config/containers/systemd/prometheus.container, reporting generated + active (`systemctl --user is-enabled
Config	/opt/prometheus/config/prometheus.yml, installed from shared/prometheus/prometheus.yml on every deploy, together with /opt/prometheus/config/rules/llm-fleet.yml from shared/prometheus/rules/llm-fleet.yml (scripts/deploy-remote.sh:219-227)
Data	/opt/prometheus/data (TSDB, prometheus.container:52)
Retention	--storage.tsdb.retention.time=30d (prometheus.container:49)
Alerting	4 alert rules + 2 recording rules (shared/prometheus/rules/llm-fleet.yml: alerts LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff, LlmEndpointDown; recordings job:llm_prefix_cache_reuse:rate5m, job:llm_spec_accept_length:rate5m), confirmed against the running server: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18) lists 4 alerting + 2 recording. LlmEndpointDown is scoped up{job=~"frank-.*"} == 0, so only the frank-* LLM jobs can page on it — any other job that is structurally never-green must stay out of that family (Stale Signal Audit). Evaluated and delivered since 2026-09-17: the alerting: block ships to alertmanager:9093 (prometheus.yml:19-22) → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
prometheus	localhost:9090	itself
node-dankclaw	host.containers.internal:9100	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
node-dankserver	192.168.0.235:9100	host metrics from Node-Exporter — a different machine, so the LAN address works
grafana, loki	grafana:3000, loki:3100	container names on the dankclaw_default network; both serve real /metrics
frank-archon	192.168.0.235:7770	the LLM fleet — a different machine, so the LAN address works (prometheus.yml:122-131); engine, metric families and their meaning live on frank-archon
frank-dragoon, frank-zealot	host.containers.internal:7770, :7771	the LLM fleet. They are addressed through the gateway because they are same-host — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (prometheus.yml:96-99, :108-120); what each endpoint exports is on frank-dragoon and frank-zealot

The frank-medium job is retired: no frank-medium job exists in the config or in /api/v1/targets (2026-10-18), and the endpoint itself left the fleet — its history is on frank-archon, which holds the port today.

No job reads down structurally any more (the three permanent-reds were retired 2026-10-21; census re-run after each deploy):

Job	Target	how
gitea	192.168.0.143:3000/metrics	metrics enabled in the quadlet (GITEA__metrics__ENABLED=true, ENABLED_HOST=0.0.0.0) — the page once said "stays off until [metrics] ENABLED = true in its own config"; the env form is what the deployer ships
services	http://192.168.0.143:8081 (SearXNG), :3005 (OpenWebUI), http://comfyui:8188	blackbox exporter probe, module http_2xx — the comfyui metrics job is retired, one owner per service

Scraping a web root is not a health check: an HTML page fails the scrape protocol (unsupported Content-Type), which is what kept those targets red beside healthy services. The classification rule and the honesty law live in Stale Signal Audit. The prober is blackbox-exporter.container (host netns, config shared/blackbox/blackbox.yml, bound 127.0.0.1:9115; Prometheus reaches it at the host.containers.internal gateway name). Plain targets stay host:port, never URLs — the probe job's URL targets are the documented exception, relabeled onto the prober.

frank-zealot reads up since 2026-10-18, and did not before. llama.cpp only answers /metrics when started with --metrics, so up{job="frank-zealot"} was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered /health 200 the whole time; the flag and its opt-out are on frank-zealot. Read lastError before believing an outage: a target can read down beside a healthy service.

Alert state is a live property, not a page fact: curl -s http://127.0.0.1:9090/api/v1/alerts returned no alerts at the 2026-10-18 morning census, one firing LlmPrefixCacheCollapse (info, frank-zealot, activeAt 11:10:48Z) minutes later, and none again by 11:56Z.

Volumes

Container path	Host path
/etc/prometheus	/opt/prometheus/config (prometheus.container:51)
/prometheus	/opt/prometheus/data (prometheus.container:52)

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on (prometheus.container:53, scripts/deploy-remote.sh — the prometheus) case of deploy_observability waits on http://127.0.0.1:9090/-/healthy).

Gotchas

• Keep the node-exporter address in step with the unit. node-dankclaw scrapes host.containers.internal:9100 (prometheus.yml:37-39), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; node-exporter.container:39 binds 127.0.0.1:9100 + 192.168.0.143:9100. Change one side without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats:108-131 seam pins the pairing in both directions.
• The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
• Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
• Set a retention policy or the data volume grows without bound (30 days here — --storage.tsdb.retention.time=30d, prometheus.container:49).
• The lifecycle endpoint is enabled (--web.enable-lifecycle, prometheus.container:49), which is why this API must never be published beyond loopback.

See Also

• Grafana — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit 6a19143, installed on the host by scripts/deploy-remote.sh:286-300)
• Node-Exporter — the host metrics source
• Observability — the whole stack, including what is still unconfigured