Files

T

Frank a61308d72e docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed

Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):

- Observability Provisioning: destroyed the 63-line 'Current delivered
  state' block — chain topology/relay/delivery restates [[Observability]]
  and [[Alertmanager]] (those are the owners). Its three sole-custody laws
  MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
  down-beside-healthy -> read lastError; Prometheus alert state in process
  vs Alertmanager's --storage.path persistence; candidate-arm metrics are
  never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
  never reuse a B/token ratio across geometries; pin pool bytes
  floor-sized), era-labelled the rates, destroyed the measured ladder
  table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.

check-okf 109/0, vault links 0 unresolved, docs seams green.

2026-10-21 02:35:44 -07:00

6.4 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability Provisioning

The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.

The config triad

Repo path	What it holds
`shared/prometheus/prometheus.yml`	scrape jobs (global + per-host)
`shared/prometheus/rules/llm-fleet.yml`	alert rules + recording rules
`shared/grafana/provisioning/`	datasources + dashboards (provisioned tree)

Targets and datasources reference container names (grafana:3000 loki:3100 — shared/prometheus/prometheus.yml:52-61), never loopback-published ports; alloy is not scraped at all (no job in the config — its 127.0.0.1:12345 UI is only probed by the homepage timer, shared/homepage/services.json:233). The LLM endpoints split by topology: the two same-host ones (frank-dragoon :7770, frank-zealot :7771) are scraped as host.containers.internal:<port> because a bridge container cannot reach a host-netns bind at the host's LAN IP (prometheus.yml:89-99), and only the endpoint on another machine (192.168.0.235:7770) is addressed by LAN IP (prometheus.yml:128-131). Prometheus 3 refuses HTML responses as metrics — HTML-scraped "probes" are structurally inert and need a blackbox exporter (see Stale Signal Audit).

The deploy path

scripts/deploy-remote.sh carries one copier per config surface, all called from deploy_observability:

copy_prometheus_config → /opt/prometheus/config/prometheus.yml and config/rules/llm-fleet.yml (:219-227)
copy_alertmanager_config → /opt/alertmanager/config/alertmanager.yml (:232-235)
install_signal_relay → the user unit + iptables + the health wait on 192.168.0.143:18093/health (:243-274)
copy_loki_config → /opt/loki/config/loki-config.yaml (:276-279)
copy_alloy_config → /opt/alloy/config/config.alloy (:281-284)
copy_grafana_provisioning → /opt/grafana/provisioning/ (:286-300)

scripts/deploy-plan.py maps shared/prometheus/** to prometheus (plus alertmanager when the changed file is alertmanager.yml, :251-256) shared/loki/** to loki and shared/alloy/** to alloy (:257-260), and shared/grafana/** to grafana (:261-264); changes under dankclaw/observability/ select the touched unit and its dependency consumer (:321-328), and a change to either relay file redeploys alertmanager (:399-402). So deploy.sh dankclaw --service grafana (etc.) restarts only what the change touched. Seam tests in tests/observability_quadlet.bats pin the unit files exist and are boot-persistent/self-healing (:36-62), that no unit publishes on a wildcard (:63-81), that the copiers install each unit (:250), that grafana provisioning is declared in git and installed (:345), and that the provisioned paths match the units' mounts.

Entry points

Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the vantage first: container name if the exporter is a container on dankclaw_default; host.containers.internal:<port> if it is a process in the host's own network namespace on this same machine (a bridge container cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only if it is a process on a different machine; a blackbox module if it is an HTTP probe, not a metrics endpoint.

Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml. Current delivered set: 4 alert rules + 2 recording rules — LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff LlmEndpointDown, plus job:llm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m (shared/prometheus/rules/llm-fleet.yml:32-110) confirmed live: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18) reports 4 alerting + 2 recording. Rules reload with the prometheus service.

Add a datasource — drop a YAML file into shared/grafana/provisioning/datasources/. Point it at the container name of the backend (http://prometheus:9090), not loopback.

Add a dashboard — drop the JSON into shared/grafana/provisioning/dashboards/. Provisioned dashboards live in unified storage, not (only) the SQLite table — grep the API, not the DB, when verifying (see the verify section of Fleet Audit and Posture). (unverified 2026-10-18: the storage-backend claim needs an admin password to confirm against grafana.db and the API side by side)

Verify through the APIs, not the filesystem

# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'

The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and up for prometheus, node-dankclaw, node-dankserver, grafana, loki frank-zealot, frank-dragoon, frank-archon with the four structural downs listed in Prometheus. The Grafana query needs the local admin password and was not run here; the filesystem side (ls /opt/grafana/provisioning/*) is proof of installation only — see the next paragraph.

Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).

Observability — stack topology and the alert delivery chain
Stale Signal Audit — classifying down targets before "fixing" them, and the down-beside-healthy / alert-state-persistence / candidate-arm rules
Quadlet Lifecycle — the units that host these services
Deploy State Triage — how the planner selects these services

6.4 KiB Raw Blame History