observability: provision Grafana from git — datasource plus the LLM dashboard

Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):

- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
  container name, not the loopback publish, with a fixed uid so dashboards
  reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
  rate, e2e latency p95, and decode/prefill throughput, with an endpoint
  variable covering both frank-medium and frank-large. Every series name was
  checked against the live /metrics endpoint: this fleet already shipped one
  comment naming a metric that does not exist, and a dashboard full of those
  would be worse.

Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.

Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
This commit is contained in:
Frank committed 2026-09-16 06:39:15 -07:00
1 parent 3037240f4f
commit 6a191430ca
6 files changed
+232

No files matched your search

+25
View File
@@ -299,3 +299,28 @@ PAIRS
grep -q 'job_name: "frank-medium"' "$cfg"
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
}
@test "grafana provisioning is declared in git and installed by the deployer" {
# Datasources belong in the repository, not in grafana.db: a UI-configured
# one is invisible to review and lost with the data volume.
local ds="$REPO_ROOT/shared/grafana/provisioning/datasources/prometheus.yml"
local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/llm-fleet.json"
local prov="$REPO_ROOT/shared/grafana/provisioning/dashboards/provider.yml"
[ -f "$ds" ] && [ -f "$dash" ] && [ -f "$prov" ]
python3 -c "import json,sys; json.load(open('$dash'))"
# Container-name address, like every other on-network scrape target: the
# published port is loopback-only and would be wrong from Grafana.
grep -q 'url: http://prometheus:9090' "$ds"
grep -q 'uid: prometheus' "$ds"
# The unit already mounts the directory; the deployer must fill it.
grep -q '/opt/grafana/provisioning:/etc/grafana/provisioning:ro' \
"$REPO_ROOT/dankclaw/observability/quadlet/grafana.container"
grep -q 'copy_grafana_provisioning' "$REPO_ROOT/scripts/deploy-remote.sh"
}
@test "the planner selects grafana for shared/grafana changes" {
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
--paths shared/grafana/provisioning/datasources/prometheus.yml
[ "$status" -eq 0 ]
[ "$output" = "grafana" ]
}