docs(observability): post-blackbox state, copier law, label-promotion why
Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.
This commit is contained in:
1 parent
078b550bd8
commit
cb86ad8fd7
3 files changed
+41
-39
No files matched your search
@@ -20,6 +20,8 @@ committed.
|
||||
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
|
||||
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
|
||||
|
||||
Targets and datasources reference **container names** (`grafana:3000`
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
|
||||
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
|
||||
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
|
||||
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
|
||||
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
|
||||
[[Stale Signal Audit]]).
|
||||
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
|
||||
deployed 2026-10-21; see [[Stale Signal Audit]]).
|
||||
|
||||
## The deploy path
|
||||
|
||||
@@ -43,7 +45,8 @@ from `deploy_observability`:
|
||||
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
|
||||
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
|
||||
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
|
||||
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
|
||||
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
|
||||
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
|
||||
|
||||
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
||||
@@ -109,8 +112,8 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
||||
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
||||
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
|
||||
listed in [[Prometheus]]. The Grafana proof is the browser session above, run
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
|
||||
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
|
||||
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
|
||||
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
|
||||
only — see *Add a dashboard* for why a repo JSON is not a live board.
|
||||
|
||||
Reference in new issue
Block a user