Files

T

Frank 139a3cd4fa fix(blackbox): unit had no Image and no Network=host - quadlet refused to generate it

The deploy died at 'Unit blackbox-exporter.service not found': a
.container without Image= is not generated at all, and the header
argued for Network=host it never declared. Completes the unit (digest
pinned prom/blackbox-exporter, Network=host) and fixes the bind
topology the first cut got wrong:

- Prometheus is bridge-networked and the netavark hairpin gap bars it
  from its own host's LAN IP, so a LAN-only prober bind is unscrapable
  and a loopback-only bind was claimed to ride the gateway on faith.
  The prober now binds 9115 on the podman gateway (scrape path) AND
  the LAN (healthcheck/diagnostics); the listen_address line leaves
  blackbox.yml so config and Exec cannot disagree.
- comfyui probe target http://comfyui:8188 -> http://127.0.0.1:8188:
  bridge container names do not resolve in the host netns; the
  published 0.0.0.0:8188 answers on loopback there.
- New seam: every observability unit must declare Image= (red on the
  shipped file by name); copier test pinned module presence instead
  of the removed listen line. Docs updated to the real topology.

2026-10-21 15:41:37 -07:00

7.7 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability Provisioning

The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.

The config triad

Repo path	What it holds
`shared/prometheus/prometheus.yml`	scrape jobs (global + per-host)
`shared/prometheus/rules/llm-fleet.yml`	alert rules + recording rules
`shared/grafana/provisioning/`	datasources + dashboards (provisioned tree)
`shared/alloy/config.alloy`	the log pipeline: journal source → label promotion → Loki push
`shared/blackbox/blackbox.yml`	prober modules (`http_2xx`); the listen bind lives on the unit's `Exec`

Targets and datasources reference container names (grafana:3000 loki:3100 — shared/prometheus/prometheus.yml:52-61), never loopback-published ports; alloy is not scraped at all (no job in the config — its 127.0.0.1:12345 UI is only probed by the homepage timer, shared/homepage/services.json:233). The LLM endpoints split by topology: the two same-host ones (frank-dragoon :7770, frank-zealot :7771) are scraped as host.containers.internal:<port> because a bridge container cannot reach a host-netns bind at the host's LAN IP (prometheus.yml:89-99), and only the endpoint on another machine (192.168.0.235:7770) is addressed by LAN IP (prometheus.yml:128-131). Prometheus 3 refuses HTML responses as metrics — HTTP liveness rides the blackbox exporter (shared/blackbox/blackbox.yml, deployed 2026-10-21; see Stale Signal Audit).

The deploy path

scripts/deploy-remote.sh carries one copier per config surface, all called from deploy_observability:

copy_prometheus_config → /opt/prometheus/config/prometheus.yml and config/rules/llm-fleet.yml (:219-227)
copy_alertmanager_config → /opt/alertmanager/config/alertmanager.yml (:232-235)
install_signal_relay → the user unit + iptables + the health wait on 192.168.0.143:18093/health (:243-274)
copy_loki_config → /opt/loki/config/loki-config.yaml (:276-279)
copy_alloy_config → /opt/alloy/config/config.alloy — substitutes the GRAFANA_HOST_LABEL token with the deploy target's name and refuses to install if the token survives; ALLOY_CONF_ROOT is the bats sandbox seam
copy_blackbox_config → /opt/blackbox/blackbox.yml (single-file mount of the prober); BLACKBOX_CONF_ROOT is its sandbox seam
copy_grafana_provisioning → /opt/grafana/provisioning/ — datasources, provider, and every dashboards/*.json (a glob, not a hand-listed subset; GRAFANA_PROV_ROOT is the test seam the bats case uses to run the copier into a sandbox)

scripts/deploy-plan.py maps shared/prometheus/** to prometheus (plus alertmanager when the changed file is alertmanager.yml, :251-256) shared/loki/** to loki and shared/alloy/** to alloy (:257-260), and shared/grafana/** to grafana (:261-264); changes under dankclaw/observability/ select the touched unit and its dependency consumer (:321-328), and a change to either relay file redeploys alertmanager (:399-402). So deploy.sh dankclaw --service grafana (etc.) restarts only what the change touched. Seam tests in tests/observability_quadlet.bats pin the unit files exist and are boot-persistent/self-healing (:36-62), that no unit publishes on a wildcard (:63-81), that the copiers install each unit (:250), that grafana provisioning is declared in git and installed (:345), and that the provisioned paths match the units' mounts.

Entry points

Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the vantage first: container name if the exporter is a container on dankclaw_default; host.containers.internal:<port> if it is a process in the host's own network namespace on this same machine (a bridge container cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only if it is a process on a different machine; a blackbox module if it is an HTTP probe, not a metrics endpoint.

Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml. Current delivered set: 4 alert rules + 2 recording rules — LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff LlmEndpointDown, plus job:llm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m (shared/prometheus/rules/llm-fleet.yml:32-110) confirmed live: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18) reports 4 alerting + 2 recording. Rules reload with the prometheus service.

Add a datasource — drop a YAML file into shared/grafana/provisioning/datasources/. Point it at the container name of the backend (http://prometheus:9090), not loopback.

Add a dashboard — drop the JSON into shared/grafana/provisioning/dashboards/; the copier globs the directory, so the next --service grafana deploy provisions it. This is a repaired law, not a given: until 2026-10-21 the copier named llm-fleet.json alone, and the containers board — merged, tested, documented as "working" — had never been provisioned; the other boards existed on disk only because someone had hand-copied them once. A repo JSON is not a live dashboard until the copier carries it, and deploy.sh cannot see the gap because it only reports the copy it performed.

Verify through the APIs, not the filesystem

# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana 13: /api/health, /api/org, /login all 404 UNAUTHENTICATED, and there
# is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only
# account is the Authentik RP login). The proven surface is a real browser
# session: warm the Anubis challenge on the vhost root, then hit
# /login/generic_oauth with the e2e storage state (tests/e2e/.auth/qa-e2e-admin.json)
# — exactly the path specs/20-grafana-sso.spec.ts drives — and read the boards.

The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and up for prometheus, node-dankclaw, node-dankserver, grafana, loki frank-zealot, frank-dragoon, frank-archon. The four structural downs that census recorded are retired (2026-10-21, Prometheus). The Grafana proof is the browser session above, run 2026-10-21: it found a provisioned-file gap the filesystem listing hid. The filesystem side (ls /opt/grafana/provisioning/*) is proof of installation only — see Add a dashboard for why a repo JSON is not a live board.

Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).

Observability — stack topology and the alert delivery chain
Stale Signal Audit — classifying down targets before "fixing" them, and the down-beside-healthy / alert-state-persistence / candidate-arm rules
Quadlet Lifecycle — the units that host these services
Deploy State Triage — how the planner selects these services

7.7 KiB Raw Blame History