docs(observability): post-blackbox state, copier law, label-promotion why

Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
This commit is contained in:
Frank committed 2026-10-21 13:33:03 -07:00
1 parent 078b550bd8
commit cb86ad8fd7
3 files changed
+41 -39

No files matched your search

@@ -20,6 +20,8 @@ committed.
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
Targets and datasources reference **container names** (`grafana:3000`
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
@@ -31,8 +33,8 @@ scraped as `host.containers.internal:<port>` because a bridge container cannot
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
[[Stale Signal Audit]]).
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
deployed 2026-10-21; see [[Stale Signal Audit]]).
## The deploy path
@@ -43,7 +45,8 @@ from `deploy_observability`:
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
@@ -109,8 +112,8 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and `up` for
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
listed in [[Prometheus]]. The Grafana proof is the browser session above, run
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
only — see *Add a dashboard* for why a repo JSON is not a live board.