fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it

The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:

- Prometheus is bridge-networked and the netavark hairpin gap bars it
  from its own host's LAN IP, so a LAN-only prober bind is unscrapable
  and a loopback-only bind was claimed to ride the gateway on faith.
  The prober now binds 9115 on the podman gateway (scrape path) AND
  the LAN (healthcheck/diagnostics); the listen_address line leaves
  blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
  bridge container names do not resolve in the host netns; the
  published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
  shipped file by name); copier test pinned module presence instead
  of the removed listen line. Docs updated to the real topology.
This commit is contained in:
Frank committed 2026-10-21 15:41:37 -07:00
1 parent 5ffbf19fb2
commit 139a3cd4fa
8 files changed
+59 -35

No files matched your search

@@ -21,7 +21,7 @@ committed.
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`); the listen bind lives on the unit's `Exec` |
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -163,6 +163,6 @@ verify it through the APIs.
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `shared/blackbox/blackbox.yml` (prober modules + loopback bind)
- `shared/blackbox/blackbox.yml` (prober modules; the listen bind is on the unit's `Exec`)
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -48,16 +48,17 @@ retired 2026-10-21; census re-run after each deploy):
| Job | Target | how |
|---|---|---|
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://127.0.0.1:8188` (ComfyUI) | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service. The prober runs in the host netns, so comfyui is probed over loopback (its published port); bridge container names do not resolve there |
Scraping a web root is not a health check: an HTML page fails the scrape
protocol (`unsupported Content-Type`), which is what kept those targets red
beside healthy services. The classification rule and the honesty law live in
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
reaches it at the `host.containers.internal` gateway name). Plain targets stay
`host:port`, never URLs — the probe job's URL targets are the documented
exception, relabeled onto the prober.
netns, config `shared/blackbox/blackbox.yml`); it binds 9115 on both the LAN
and the podman gateway, and Prometheus scrapes the gateway address — the
netavark hairpin gap bars this container from its own host's LAN IP.
Plain targets stay `host:port`, never URLs — the probe job's URL targets are
the documented exception, relabeled onto the prober.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.