serverAppConfigs/serverAppConfigs-docs/Infrastructure/Prometheus.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages.	infrastructure prometheus	2026-09-21

Prometheus

Prometheus collects metrics from the services listed below by scraping their /metrics endpoints, and it is the datasource behind the provisioned llm-fleet dashboard in Grafana (datasource uid prometheus, shared/grafana/provisioning/datasources/prometheus.yml:14-21). node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	docker.io/prom/prometheus:latest
Published	127.0.0.1:9090 → 9090, loopback only — never a wildcard (dankclaw/observability/quadlet/prometheus.container:50; ss -ltnp 2026-10-18 shows 127.0.0.1:9090 and no other bind; tests/observability_quadlet.bats:63 pins it)
Unit	dankclaw/observability/quadlet/prometheus.container → prometheus.service; Restart=always/RestartSec=10 + WantedBy=default.target (prometheus.container:59-64), installed at ~/.config/containers/systemd/prometheus.container, reporting generated + active (`systemctl --user is-enabled
Config	/opt/prometheus/config/prometheus.yml, installed from shared/prometheus/prometheus.yml on every deploy, together with /opt/prometheus/config/rules/llm-fleet.yml from shared/prometheus/rules/llm-fleet.yml (scripts/deploy-remote.sh:219-227)
Data	/opt/prometheus/data (TSDB, prometheus.container:52)
Retention	--storage.tsdb.retention.time=30d (prometheus.container:49)
Alerting	4 alert rules + 2 recording rules (shared/prometheus/rules/llm-fleet.yml: alerts LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff, LlmEndpointDown; recordings job:llm_prefix_cache_reuse:rate5m, job:llm_spec_accept_length:rate5m), confirmed against the running server: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18) lists 4 alerting + 2 recording. Evaluated and delivered since 2026-09-17: the alerting: block ships to alertmanager:9093 (prometheus.yml:19-22) → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
prometheus	localhost:9090	itself
node-dankclaw	host.containers.internal:9100	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
node-dankserver	192.168.0.235:9100	host metrics from Node-Exporter — a different machine, so the LAN address works
grafana, loki	grafana:3000, loki:3100	container names on the dankclaw_default network; both serve real /metrics
frank-archon	192.168.0.235:7770	the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus radiance:* (prometheus.yml:122-131); a different machine, so the LAN address works
frank-dragoon, frank-zealot	host.containers.internal:7770, :7771	the LLM fleet — both export the llamacpp: namespace (halogen included: curl -s http://127.0.0.1:7770/metrics → llamacpp:prompt_tokens_total, text/plain; version=0.0.4, 2026-10-18). They are addressed through the gateway because they are same-host — frank-zealot is a host-process bind, frank-dragoon a container publishing 0.0.0.0:7770->8731, not a net=host pair (prometheus.yml:96-99, :108-120)

The frank-medium job and its vllm: namespace are retired: that endpoint left the fleet, frank-archon briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (prometheus.yml:101-107). No frank-medium job exists in the config or in /api/v1/targets (2026-10-18).

Three jobs read down today, and they are not outages (re-audited against curl -s http://127.0.0.1:9090/api/v1/targets, 2026-10-18; first audited 2026-09-16):

Job	Target	health	lastError, verbatim
gitea	http://192.168.0.143:3000/metrics	down	server returned HTTP status 404 Not Found
comfyui	http://comfyui:8188/	down	received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target
services	http://192.168.0.143:8081/	down	received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target
services	http://192.168.0.143:3005/	down	same unsupported Content-Type "text/html; charset=utf-8" error

Gitea's exporter stays off until [metrics] ENABLED = true in its own config (no metrics env is set in the container, and curl -s http://127.0.0.1:3000/metrics → 404, 2026-10-18); ComfyUI and the two services probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: prometheus, node-dankclaw, node-dankserver, grafana, loki, frank-zealot, frank-dragoon, frank-archon.

frank-zealot reads up since 2026-10-18, and did not before. llama.cpp answers 501 Not Implemented on /metrics unless started with --metrics; the endpoint had no such flag, so up{job="frank-zealot"} was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered /health 200 the whole time. The METRICS=1 knob in scripts/frank-zealot.env is the fix, and the target is up as measured 2026-10-18. Read lastError before believing an outage: a target can read down beside a healthy service.

Alert state is a live property, not a page fact: curl -s http://127.0.0.1:9090/api/v1/alerts returned no alerts at the 2026-10-18 morning census, one firing LlmPrefixCacheCollapse (info, frank-zealot, activeAt 11:10:48Z) minutes later, and none again by 11:56Z.

A scrape cannot express "is this HTTP service answering?" — that is what a probe is for, and the remedy is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives the guarded vhosts a real uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
/etc/prometheus	/opt/prometheus/config (prometheus.container:51)
/prometheus	/opt/prometheus/data (prometheus.container:52)

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on (prometheus.container:53, scripts/deploy-remote.sh — the prometheus) case of deploy_observability waits on http://127.0.0.1:9090/-/healthy).

Gotchas

• Keep the node-exporter address in step with the unit. node-dankclaw scrapes host.containers.internal:9100 (prometheus.yml:37-39), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; node-exporter.container:39 binds 127.0.0.1:9100 + 192.168.0.143:9100. Change one side without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats:108-131 seam pins the pairing in both directions.
• The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
• Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
• Set a retention policy or the data volume grows without bound (30 days here — --storage.tsdb.retention.time=30d, prometheus.container:49).
• The lifecycle endpoint is enabled (--web.enable-lifecycle, prometheus.container:49), which is why this API must never be published beyond loopback.

See Also

• Grafana — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit 6a19143, installed on the host by scripts/deploy-remote.sh:286-300)
• Node-Exporter — the host metrics source
• Observability — the whole stack, including what is still unconfigured