0 Fork 0

Files

T

Frank 135bb2194d agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set

2026-09-17 12:27:08 -07:00

4.9 KiB

Raw Blame History

name, description

name	description
grafana-prometheus-provisioning-as-code	Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives.

Observability as code (Grafana + Prometheus, Dankclaw fleet)

Use when adding a scrape target, an alert rule, a datasource, or a dashboard to this homelab. Everything lives in git; Grafana and Prometheus only ever read what the deployer installs.

The triad — every change touches three places

Config in the repo
- shared/prometheus/prometheus.yml — scrape_configs, and rule_files: pointing at /etc/prometheus/rules/*.yml
- shared/prometheus/rules/<name>.yml — alert/recording rules
- shared/grafana/provisioning/datasources/prometheus.yml — datasource
- shared/grafana/provisioning/dashboards/{provider.yml,<name>.json}
The copier in scripts/deploy-remote.sh — copy_prometheus_config installs config and rules into /opt/prometheus/config/ (the container mounts that to /etc/prometheus); copy_grafana_provisioning installs into /opt/grafana/provisioning/ (the grafana unit already mounts it read-only). Paths are absolute and run_privileged.
The planner in scripts/deploy-plan.py — a shared/<group>/** change must select the service that reads it (shared/prometheus → prometheus, shared/grafana → grafana). Missing mapping = the file deploys nowhere and nothing reports an error.

Add a seam in tests/observability_quadlet.bats for each change: the file exists, the config references it, the deployer copies it, and — where two files must agree (target vs published port) — that they do.

Reach targets by container name, never the loopback publish

Datasources use http://prometheus:9090 and scrape jobs for co-located services use prometheus:9090 / loki:3100. Published ports are 127.0.0.1-only by design (see the exposure rules), so a browser-facing URL or a published port in a datasource is wrong from inside the network. LLM endpoints are the exception: they are on other hosts, so use their LAN address.

Three failure modes that produce no error message

A rule file nothing loads evaluates nothing. Prometheus reports no problem for a rule_files entry that matches nothing, or for a rules file nobody references. Ship both together and assert the reference in a seam.
A provisioned dashboard is not in the legacy dashboard table on the current Grafana. Reading grafana.db for it returns empty and looks like a failed provision. It lives in unified storage behind dashboard.grafana.app; query the API instead.
Lifetime averages are not per-request evidence. e.g. an engine's "Prefix cache hit rate: 29 %" is every prompt it ever saw, including benchmarks. Measure the behaviour you care about with a targeted probe.

Verify through the channel the consumer uses

# Prometheus: config and rules parse/load
ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/<name>.yml'
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \
  | python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"'
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...'   # every target health=up
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...'                  # 0 firing when healthy

# Grafana: provisioned objects and a live query (admin/admin on this install)
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"'
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/<uid>"'
ssh dankclaw 'curl -s -u admin:admin \
  "http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"'

A target that is up proves reachability, not data: also run one query for a series you expect and confirm it returns a sample.

Grafana facts for this install

No UI-configured state: editable: false on the datasource, fixed uid: prometheus so dashboards reference it without a lookup.
Dashboards re-read from disk every 30 s (updateIntervalSeconds), so a JSON edit appears without a restart — but the file must still be installed, which means a deploy.
There is no Alertmanager and no contact point: rules evaluate, alerts show on /alerts, in the ALERTS metric and on a dashboard panel, and reach nobody. Say that plainly in docs rather than implying observability.

Naming metrics

Never write a series name from memory into a config, a rule, or a dashboard — one shipped comment here named a metric that does not exist. Dump the live names first and paste them:

ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u'

4.9 KiB Raw Blame History