Frank
cb86ad8fd7
Prometheus/Observability/Provisioning pages: every job up after the blackbox swap, the copier list learns copy_blackbox_config and the alloy validate gate, the config table gains alloy + blackbox rows, and the logs bullet states the stream-label law instead of the retired 'awaiting deploy' note.
7.7 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||||
|---|---|---|---|---|---|---|---|---|---|
| reference | Observability Provisioning | Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands. |
|
2026-10-21T00:00:00-07:00 |
Observability Provisioning
The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.
The config triad
| Repo path | What it holds |
|---|---|
shared/prometheus/prometheus.yml |
scrape jobs (global + per-host) |
shared/prometheus/rules/llm-fleet.yml |
alert rules + recording rules |
shared/grafana/provisioning/ |
datasources + dashboards (provisioned tree) |
shared/alloy/config.alloy |
the log pipeline: journal source → label promotion → Loki push |
shared/blackbox/blackbox.yml |
prober modules (http_2xx) + the loopback bind |
Targets and datasources reference container names (grafana:3000
loki:3100 — shared/prometheus/prometheus.yml:52-61), never
loopback-published ports; alloy is not scraped at all (no job in the
config — its 127.0.0.1:12345 UI is only probed by the homepage timer,
shared/homepage/services.json:233). The LLM endpoints split by topology: the
two same-host ones (frank-dragoon :7770, frank-zealot :7771) are
scraped as host.containers.internal:<port> because a bridge container cannot
reach a host-netns bind at the host's LAN IP (prometheus.yml:89-99), and only
the endpoint on another machine (192.168.0.235:7770) is addressed by LAN IP
(prometheus.yml:128-131). Prometheus 3 refuses HTML responses as metrics —
HTTP liveness rides the blackbox exporter (shared/blackbox/blackbox.yml,
deployed 2026-10-21; see Stale Signal Audit).
The deploy path
scripts/deploy-remote.sh carries one copier per config surface, all called
from deploy_observability:
copy_prometheus_config→/opt/prometheus/config/prometheus.ymlandconfig/rules/llm-fleet.yml(:219-227)copy_alertmanager_config→/opt/alertmanager/config/alertmanager.yml(:232-235)install_signal_relay→ the user unit + iptables + the health wait on192.168.0.143:18093/health(:243-274)copy_loki_config→/opt/loki/config/loki-config.yaml(:276-279)copy_alloy_config→/opt/alloy/config/config.alloy— substitutes theGRAFANA_HOST_LABELtoken with the deploy target's name and refuses to install if the token survives;ALLOY_CONF_ROOTis the bats sandbox seamcopy_blackbox_config→/opt/blackbox/blackbox.yml(single-file mount of the prober);BLACKBOX_CONF_ROOTis its sandbox seamcopy_grafana_provisioning→/opt/grafana/provisioning/— datasources, provider, and everydashboards/*.json(a glob, not a hand-listed subset;GRAFANA_PROV_ROOTis the test seam the bats case uses to run the copier into a sandbox)
scripts/deploy-plan.py maps shared/prometheus/** to prometheus (plus
alertmanager when the changed file is alertmanager.yml, :251-256)
shared/loki/** to loki and shared/alloy/** to alloy (:257-260), and
shared/grafana/** to grafana (:261-264); changes under
dankclaw/observability/ select the touched unit and its dependency consumer
(:321-328), and a change to either relay file redeploys alertmanager
(:399-402). So deploy.sh dankclaw --service grafana (etc.) restarts only
what the change touched. Seam tests in tests/observability_quadlet.bats pin
the unit files exist and are boot-persistent/self-healing (:36-62), that no
unit publishes on a wildcard (:63-81), that the copiers install each unit
(:250), that grafana provisioning is declared in git and installed
(:345), and that the provisioned paths match the units' mounts.
Entry points
Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the
vantage first: container name if the exporter is a container on
dankclaw_default; host.containers.internal:<port> if it is a process in the
host's own network namespace on this same machine (a bridge container
cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only
if it is a process on a different machine; a blackbox module if it is an
HTTP probe, not a metrics endpoint.
Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml.
Current delivered set: 4 alert rules + 2 recording rules —
LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff
LlmEndpointDown, plus job:llm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m (shared/prometheus/rules/llm-fleet.yml:32-110)
confirmed live: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18)
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
Add a datasource — drop a YAML file into
shared/grafana/provisioning/datasources/. Point it at the container name of
the backend (http://prometheus:9090), not loopback.
Add a dashboard — drop the JSON into
shared/grafana/provisioning/dashboards/; the copier globs the directory, so
the next --service grafana deploy provisions it. This is a repaired law, not
a given: until 2026-10-21 the copier named llm-fleet.json alone, and the
containers board — merged, tested, documented as "working" — had never
been provisioned; the other boards existed on disk only because someone had
hand-copied them once. A repo JSON is not a live dashboard until the copier
carries it, and deploy.sh cannot see the gap because it only reports the
copy it performed.
Verify through the APIs, not the filesystem
# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
# account is the Authentik RP login). The proven surface is a real browser
# session: warm the Anubis challenge on the vhost root, then hit
# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.
The two Prometheus queries were run on 2026-10-18 and are what every number on
this page comes from: 4 alerting + 2 recording rules, and up for
prometheus, node-dankclaw, node-dankserver, grafana, loki
frank-zealot, frank-dragoon, frank-archon. The four structural downs
that census recorded are retired (2026-10-21, Prometheus). The Grafana proof is the browser session above, run
2026-10-21: it found a provisioned-file gap the filesystem listing hid. The
filesystem side (ls /opt/grafana/provisioning/*) is proof of installation
only — see Add a dashboard for why a repo JSON is not a live board.
Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).
Related
- Observability — stack topology and the alert delivery chain
- Stale Signal Audit — classifying down targets before "fixing" them, and the down-beside-healthy / alert-state-persistence / candidate-arm rules
- Quadlet Lifecycle — the units that host these services
- Deploy State Triage — how the planner selects these services