docs(observability): post-blackbox state, copier law, label-promotion why
Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.
This commit is contained in:
1 parent
078b550bd8
commit
cb86ad8fd7
3 files changed
+41
-39
No files matched your search
@@ -42,29 +42,27 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
|
||||
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
|
||||
|
||||
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
|
||||
**No job reads down structurally any more** (the three permanent-reds were
|
||||
retired 2026-10-21; census re-run after each deploy):
|
||||
|
||||
| Job | Target | `health` | `lastError`, verbatim |
|
||||
|-----|--------|----------|------------------------|
|
||||
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
|
||||
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
|
||||
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
|
||||
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
|
||||
| Job | Target | how |
|
||||
|---|---|---|
|
||||
| `gitea` | `192.168.0.143:3000/metrics` | metrics enabled in the quadlet (`GITEA__metrics__ENABLED=true`, `ENABLED_HOST=0.0.0.0`) — the page once said "stays off until `[metrics] ENABLED = true` in its own config"; the env form is what the deployer ships |
|
||||
| `services` | `http://192.168.0.143:8081` (SearXNG), `:3005` (OpenWebUI), `http://comfyui:8188` | **blackbox exporter** probe, module `http_2xx` — the comfyui metrics job is retired, one owner per service |
|
||||
|
||||
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
|
||||
Scraping a web root is not a health check: an HTML page fails the scrape
|
||||
protocol (`unsupported Content-Type`), which is what kept those targets red
|
||||
beside healthy services. The classification rule and the honesty law live in
|
||||
[[Stale Signal Audit]]. The prober is `blackbox-exporter.container` (host
|
||||
netns, config `shared/blackbox/blackbox.yml`, bound 127.0.0.1:9115; Prometheus
|
||||
reaches it at the `host.containers.internal` gateway name). Plain targets stay
|
||||
`host:port`, never URLs — the probe job's URL targets are the documented
|
||||
exception, relabeled onto the prober.
|
||||
|
||||
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
||||
|
||||
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
|
||||
|
||||
The remedy for a probe-shaped question is the **blackbox exporter**; the
|
||||
classification rule and the honesty law for it live in [[Stale Signal Audit]].
|
||||
Until the exporter is deployed, these three stay red for a structural reason:
|
||||
deleting them would
|
||||
hide a real service, and fixing them by publishing databases or enabling metrics
|
||||
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
||||
URLs.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|
||||
Reference in new issue
Block a user