Frank
139a3cd4fa
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
91 lines
8.1 KiB
Markdown
91 lines
8.1 KiB
Markdown
---
|
|
type: reference
|
|
description: "Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural."
|
|
tags:
|
|
- infrastructure
|
|
- prometheus
|
|
timestamp: 2026-09-21
|
|
---
|
|
|
|
# Prometheus
|
|
|
|
Prometheus collects metrics from the services listed below by scraping their `/metrics` endpoints, and it is the datasource behind the provisioned `llm-fleet` dashboard in [[Grafana]] (datasource uid `prometheus`, `shared/grafana/provisioning/datasources/prometheus.yml:14-21`). node-exporter supplies host metrics from both machines.
|
|
|
|
**Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
|
|
Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
|
**Port:** 9090 (loopback) — **Docs:** https://prometheus.io/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/prom/prometheus:latest` |
|
|
| Published | `127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
|
|
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active prometheus` 2026-10-18) |
|
|
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
|
|
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
|
|
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
|
|
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
|
|
|
|
## Targets
|
|
|
|
`shared/prometheus/prometheus.yml` is the source of truth; the deploy copies it into the container's `/etc/prometheus/`. In outline:
|
|
|
|
| Job | Target | Why |
|
|
|-----|--------|-----|
|
|
| `prometheus` | `localhost:9090` | itself |
|
|
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
|
|
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
|
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
|
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
|
|
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
|
|
|
|
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
|
|
|
|
**No job reads down structurally any more** (the three permanent-reds were
|
|
retired 2026-10-21; census re-run after each deploy):
|
|
|
|
| Job | Target | how |
|
|
|---|---|---|
|
|
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
|
|
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
|
|
|
|
Scraping a web root is not a health check: an HTML page fails the scrape
|
|
protocol (`unsupported Content-Type`), which is what kept those targets red
|
|
beside healthy services. The classification rule and the honesty law live in
|
|
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
|
|
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
|
|
and the podman gateway, and Prometheus scrapes the gateway address — the
|
|
netavark hairpin gap bars this container from its own host's LAN IP.
|
|
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
|
|
the documented exception, relabeled onto the prober.
|
|
|
|
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
|
|
|
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/etc/prometheus` | `/opt/prometheus/config` (`prometheus.container:51`) |
|
|
| `/prometheus` | `/opt/prometheus/data` (`prometheus.container:52`) |
|
|
|
|
## Healthcheck
|
|
|
|
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on (`prometheus.container:53`, `scripts/deploy-remote.sh` — the `prometheus)` case of `deploy_observability` waits on `http://127.0.0.1:9090/-/healthy`).
|
|
|
|
## Gotchas
|
|
|
|
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `host.containers.internal:9100` (`prometheus.yml:37-39`), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; `node-exporter.container:39` binds `127.0.0.1:9100` + `192.168.0.143:9100`. Change one side without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats:108-131` seam pins the pairing in both directions.
|
|
- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
|
|
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
|
|
- Set a retention policy or the data volume grows without bound (30 days here — `--storage.tsdb.retention.time=30d`, `prometheus.container:49`).
|
|
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`, `prometheus.container:49`), which is why this API must never be published beyond loopback.
|
|
|
|
## See Also
|
|
|
|
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit `6a19143`, installed on the host by `scripts/deploy-remote.sh:286-300`)
|
|
- [[Node-Exporter]] — the host metrics source
|
|
- [[Observability]] — the whole stack, including what is still unconfigured
|