fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
This commit is contained in:
1 parent
5ffbf19fb2
commit
139a3cd4fa
8 files changed
+59
-35
No files matched your search
@@ -21,7 +21,7 @@ committed.
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
|
||||
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
|
||||
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
|
||||
|
||||
Targets and datasources reference **container names** (`grafana:3000`
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
|
||||
@@ -163,6 +163,6 @@ verify it through the APIs.
|
||||
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
|
||||
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
|
||||
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
|
||||
- `shared/blackbox/blackbox.yml` (prober modules + loopback bind)
|
||||
- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
|
||||
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
|
||||
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
|
||||
@@ -48,16 +48,17 @@ retired 2026-10-21; census re-run after each deploy):
|
||||
| Job | Target | how |
|
||||
|---|---|---|
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
|
||||
|
||||
Scraping a web root is not a health check: an HTML page fails the scrape
|
||||
protocol (`unsupported Content-Type`), which is what kept those targets red
|
||||
beside healthy services. The classification rule and the honesty law live in
|
||||
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
|
||||
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
|
||||
reaches it at the `host.containers.internal` gateway name). Plain targets stay
|
||||
`host:port`, never URLs — the probe job's URL targets are the documented
|
||||
exception, relabeled onto the prober.
|
||||
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
|
||||
and the podman gateway, and Prometheus scrapes the gateway address — the
|
||||
netavark hairpin gap bars this container from its own host's LAN IP.
|
||||
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
|
||||
the documented exception, relabeled onto the prober.
|
||||
|
||||
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
||||
|
||||
|
||||
Reference in new issue
Block a user