feat(observability): blackbox exporter replaces the HTML-probe jobs
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe jobs were structurally dead targets reading as outages. HTTP liveness now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and ComfyUI via __param_target relabeling. The unit is a boot-persistent, self-healing root quadlet.
This commit is contained in:
1 parent
425994515a
commit
f073b91033
4 files changed
+80
-11
No files matched your search
@@ -60,20 +60,31 @@ scrape_configs:
|
||||
- targets: ["loki:3100"]
|
||||
metrics_path: "/metrics"
|
||||
|
||||
# ComfyUI (no native /metrics; probe the UI)
|
||||
- job_name: "comfyui"
|
||||
metrics_path: "/"
|
||||
static_configs:
|
||||
- targets: ["comfyui:8188"]
|
||||
|
||||
# Blackbox-style probes for service up/down (published LAN ports;
|
||||
# gitea and comfyui are covered by their own jobs above)
|
||||
# HTTP liveness, probed honestly through the blackbox exporter. Scraping a
|
||||
# web root is NOT a health check — an HTML front page fails the scrape
|
||||
# protocol (`unsupported Content-Type`), which is exactly why the old
|
||||
# `services` and `comfyui` jobs were red forever beside healthy services
|
||||
# (Stale Signal Audit §1). The prober is a host-netns container reached at
|
||||
# the podman gateway name (HostAddr= on the prometheus unit); it in turn
|
||||
# reaches both the host LAN IP and bridge container names. comfyui's UI probe
|
||||
# lives here now — its own metrics-scrape job is retired, one owner per
|
||||
# service.
|
||||
- job_name: "services"
|
||||
metrics_path: "/"
|
||||
metrics_path: /probe
|
||||
params:
|
||||
module: [http_2xx]
|
||||
static_configs:
|
||||
- targets:
|
||||
- "192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "192.168.0.143:3005" # OpenWebUI
|
||||
- "http://192.168.0.143:8081" # SearXNG (via the Gluetun tunnel)
|
||||
- "http://192.168.0.143:3005" # OpenWebUI
|
||||
- "http://comfyui:8188" # ComfyUI UI (gfx1151 generator)
|
||||
relabel_configs:
|
||||
- source_labels: [__address__]
|
||||
target_label: __param_target
|
||||
- source_labels: [__param_target]
|
||||
target_label: instance
|
||||
- target_label: __address__
|
||||
replacement: "host.containers.internal:9115"
|
||||
|
||||
# LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
|
||||
# this container, not the guarded vhosts: these are machine clients and
|
||||
|
||||
Reference in new issue
Block a user