Frank
cb86ad8fd7
Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.
133 lines
7.7 KiB
Markdown
133 lines
7.7 KiB
Markdown
---
|
|
type: reference
|
|
title: Observability Provisioning
|
|
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
|
tags: [observability, prometheus, grafana, provisioning, deployment]
|
|
timestamp: 2026-10-21T00:00:00-07:00
|
|
---
|
|
|
|
# Observability Provisioning
|
|
|
|
The monitoring stack is **provisioned as code**: dashboards, datasources,
|
|
scrape jobs and alert rules live in the repo and ride the normal deploy flow.
|
|
No hand-edits inside containers — they are wiped on recreate and never
|
|
committed.
|
|
|
|
## The config triad
|
|
|
|
| Repo path | What it holds |
|
|
|---|---|
|
|
| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
|
|
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
|
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
|
| `shared/alloy/config.alloy` | the log pipeline: journal source → label promotion → Loki push |
|
|
| `shared/blackbox/blackbox.yml` | prober modules (`http_2xx`) + the loopback bind |
|
|
|
|
Targets and datasources reference **container names** (`grafana:3000`
|
|
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
|
loopback-published ports; `alloy` is **not scraped at all** (no job in the
|
|
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
|
|
`shared/homepage/services.json:233`). The LLM endpoints split by topology: the
|
|
two same-host ones (`frank-dragoon` `:7770`, `frank-zealot` `:7771`) are
|
|
scraped as `host.containers.internal:<port>` because a bridge container cannot
|
|
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
|
|
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
|
|
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
|
|
HTTP liveness rides the **blackbox exporter** (`shared/blackbox/blackbox.yml`,
|
|
deployed 2026-10-21; see [[Stale Signal Audit]]).
|
|
|
|
## The deploy path
|
|
|
|
`scripts/deploy-remote.sh` carries one copier per config surface, all called
|
|
from `deploy_observability`:
|
|
|
|
- `copy_prometheus_config` → `/opt/prometheus/config/prometheus.yml` **and** `config/rules/llm-fleet.yml` (`:219-227`)
|
|
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
|
|
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
|
|
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
|
|
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` — substitutes the `GRAFANA_HOST_LABEL` token with the deploy target's name and **refuses to install** if the token survives; `ALLOY_CONF_ROOT` is the bats sandbox seam
|
|
- `copy_blackbox_config` → `/opt/blackbox/blackbox.yml` (single-file mount of the prober); `BLACKBOX_CONF_ROOT` is its sandbox seam
|
|
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` — datasources, provider, and **every** `dashboards/*.json` (a glob, not a hand-listed subset; `GRAFANA_PROV_ROOT` is the test seam the bats case uses to run the copier into a sandbox)
|
|
|
|
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
|
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
|
|
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
|
|
`shared/grafana/**` to `grafana` (`:261-264`); changes under
|
|
`dankclaw/observability/` select the touched unit and its dependency consumer
|
|
(`:321-328`), and a change to either relay file redeploys `alertmanager`
|
|
(`:399-402`). So `deploy.sh dankclaw --service grafana` (etc.) restarts only
|
|
what the change touched. Seam tests in `tests/observability_quadlet.bats` pin
|
|
the unit files exist and are boot-persistent/self-healing (`:36-62`), that no
|
|
unit publishes on a wildcard (`:63-81`), that the copiers install each unit
|
|
(`:250`), that grafana provisioning is declared in git and installed
|
|
(`:345`), and that the provisioned paths match the units' mounts.
|
|
|
|
## Entry points
|
|
|
|
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
|
vantage first: container name if the exporter is a container on
|
|
`dankclaw_default`; `host.containers.internal:<port>` if it is a process in the
|
|
host's own network namespace **on this same machine** (a bridge container
|
|
cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only
|
|
if it is a process on a **different** machine; a blackbox module if it is an
|
|
HTTP probe, not a metrics endpoint.
|
|
|
|
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
|
Current delivered set: **4 alert rules + 2 recording rules** —
|
|
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
|
|
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
|
|
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
|
|
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
|
|
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
|
|
|
|
**Add a datasource** — drop a YAML file into
|
|
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
|
the backend (`http://prometheus:9090`), not loopback.
|
|
|
|
**Add a dashboard** — drop the JSON into
|
|
`shared/grafana/provisioning/dashboards/`; the copier globs the directory, so
|
|
the next `--service grafana` deploy provisions it. This is a repaired law, not
|
|
a given: until 2026-10-21 the copier named `llm-fleet.json` alone, and the
|
|
`containers` board — merged, tested, documented as "working" — had **never
|
|
been provisioned**; the other boards existed on disk only because someone had
|
|
hand-copied them once. A repo JSON is not a live dashboard until the copier
|
|
carries it, and `deploy.sh` cannot see the gap because it only reports the
|
|
copy it performed.
|
|
|
|
## Verify through the APIs, not the filesystem
|
|
|
|
```bash
|
|
# prometheus: rules and targets actually loaded
|
|
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
|
|
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
|
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
|
|
# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
|
|
# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
|
|
# account is the Authentik RP login). The proven surface is a real browser
|
|
# session: warm the Anubis challenge on the vhost root, then hit
|
|
# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
|
|
# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
|
|
```
|
|
|
|
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
|
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
|
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
|
|
`frank-zealot`, `frank-dragoon`, `frank-archon`. The four structural downs
|
|
that census recorded are retired (2026-10-21, [[Prometheus]]). The Grafana proof is the browser session above, run
|
|
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
|
|
filesystem side (`ls /opt/grafana/provisioning/*`) is proof of installation
|
|
only — see *Add a dashboard* for why a repo JSON is not a live board.
|
|
|
|
Health answers from a stale process and provisioned resources can hide in
|
|
unified storage — a file existing on disk is not proof the running server
|
|
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
|
table (refused = outage, 404/HTML = config defect).
|
|
|
|
## Related
|
|
|
|
- [[Observability]] — stack topology and the alert delivery chain
|
|
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
|
|
down-beside-healthy / alert-state-persistence / candidate-arm rules
|
|
- [[Quadlet Lifecycle]] — the units that host these services
|
|
- [[Deploy State Triage]] — how the planner selects these services
|