fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it

The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:

- Prometheus is bridge-networked and the netavark hairpin gap bars it
  from its own host's LAN IP, so a LAN-only prober bind is unscrapable
  and a loopback-only bind was claimed to ride the gateway on faith.
  The prober now binds 9115 on the podman gateway (scrape path) AND
  the LAN (healthcheck/diagnostics); the listen_address line leaves
  blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
  bridge container names do not resolve in the host netns; the
  published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
  shipped file by name); copier test pinned module presence instead
  of the removed listen line. Docs updated to the real topology.
This commit is contained in:
Frank committed 2026-10-21 15:41:37 -07:00
1 parent 5ffbf19fb2
commit 139a3cd4fa
8 files changed
+59 -35

No files matched your search

@@ -48,16 +48,17 @@ retired 2026-10-21; census re-run after each deploy):
| Job | Target | how |
|---|---|---|
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
Scraping a web root is not a health check: an HTML page fails the scrape
protocol (`unsupported Content-Type`), which is what kept those targets red
beside healthy services. The classification rule and the honesty law live in
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
reaches it at the `host.containers.internal` gateway name). Plain targets stay
`host:port`, never URLs — the probe job's URL targets are the documented
exception, relabeled onto the prober.
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
and the podman gateway, and Prometheus scrapes the gateway address — the
netavark hairpin gap bars this container from its own host's LAN IP.
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
the documented exception, relabeled onto the prober.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.