fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it
The deploy died at 'Unit blackbox-exporter.service not found': a .container without Image= is not generated at all, and the header argued for Network=host it never declared. Completes the unit (digest pinned prom/blackbox-exporter, Network=host) and fixes the bind topology the first cut got wrong: - Prometheus is bridge-networked and the netavark hairpin gap bars it from its own host's LAN IP, so a LAN-only prober bind is unscrapable and a loopback-only bind was claimed to ride the gateway on faith. The prober now binds 9115 on the podman gateway (scrape path) AND the LAN (healthcheck/diagnostics); the listen_address line leaves blackbox.yml so config and Exec cannot disagree. - comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188: bridge container names do not resolve in the host netns; the published 0.0.0.0:8188 answers on loopback there. - New seam: every observability unit must declare Image= (red on the shipped file by name); copier test pinned module presence instead of the removed listen line. Docs updated to the real topology.
This commit is contained in:
1 parent
5ffbf19fb2
commit
139a3cd4fa
8 files changed
+59
-35
No files matched your search
@@ -48,16 +48,17 @@ retired 2026-10-21; census re-run after each deploy):
|
||||
| Job | Target | how |
|
||||
|---|---|---|
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
|
||||
|
||||
Scraping a web root is not a health check: an HTML page fails the scrape
|
||||
protocol (`unsupported Content-Type`), which is what kept those targets red
|
||||
beside healthy services. The classification rule and the honesty law live in
|
||||
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
|
||||
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
|
||||
reaches it at the `host.containers.internal` gateway name). Plain targets stay
|
||||
`host:port`, never URLs — the probe job's URL targets are the documented
|
||||
exception, relabeled onto the prober.
|
||||
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
|
||||
and the podman gateway, and Prometheus scrapes the gateway address — the
|
||||
netavark hairpin gap bars this container from its own host's LAN IP.
|
||||
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
|
||||
the documented exception, relabeled onto the prober.
|
||||
|
||||
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
||||
|
||||
|
||||
Reference in new issue
Block a user