observability: provision Grafana from git — datasource plus the LLM dashboard
Grafana has been running with zero datasources and zero dashboards, so the metrics scraped last commit were only reachable through the Prometheus API. This wires the plumbing and one dashboard, both declared in shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the deployer (the unit already mounted that directory read-only, so nothing about the container changes): - datasources/prometheus.yml — Prometheus at http://prometheus:9090, the container name, not the loopback publish, with a fixed uid so dashboards reference it without a lookup. - dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse rate, e2e latency p95, and decode/prefill throughput, with an endpoint variable covering both frank-medium and frank-large. Every series name was checked against the live /metrics endpoint: this fleet already shipped one comment naming a metric that does not exist, and a dashboard full of those would be worse. Dashboards stay editable in the UI (allowUiUpdates) because the panels are a starting point, but the file is the source of truth and the deployer overwrites Grafana's copy on every deploy. Planner and seams follow the existing pattern: shared/grafana/** selects the grafana service, and the tests pin the container-name address plus the provisioning contract so a rename cannot silently empty the dashboard.
This commit is contained in:
1 parent
3037240f4f
commit
6a191430ca
6 files changed
+232
No files matched your search
@@ -299,3 +299,28 @@ PAIRS
|
||||
grep -q 'job_name: "frank-medium"' "$cfg"
|
||||
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
|
||||
}
|
||||
|
||||
@test "grafana provisioning is declared in git and installed by the deployer" {
|
||||
# Datasources belong in the repository, not in grafana.db: a UI-configured
|
||||
# one is invisible to review and lost with the data volume.
|
||||
local ds="$REPO_ROOT/shared/grafana/provisioning/datasources/prometheus.yml"
|
||||
local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/llm-fleet.json"
|
||||
local prov="$REPO_ROOT/shared/grafana/provisioning/dashboards/provider.yml"
|
||||
[ -f "$ds" ] && [ -f "$dash" ] && [ -f "$prov" ]
|
||||
python3 -c "import json,sys; json.load(open('$dash'))"
|
||||
# Container-name address, like every other on-network scrape target: the
|
||||
# published port is loopback-only and would be wrong from Grafana.
|
||||
grep -q 'url: http://prometheus:9090' "$ds"
|
||||
grep -q 'uid: prometheus' "$ds"
|
||||
# The unit already mounts the directory; the deployer must fill it.
|
||||
grep -q '/opt/grafana/provisioning:/etc/grafana/provisioning:ro' \
|
||||
"$REPO_ROOT/dankclaw/observability/quadlet/grafana.container"
|
||||
grep -q 'copy_grafana_provisioning' "$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
}
|
||||
|
||||
@test "the planner selects grafana for shared/grafana changes" {
|
||||
run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
|
||||
--paths shared/grafana/provisioning/datasources/prometheus.yml
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "grafana" ]
|
||||
}
|
||||
Reference in new issue
Block a user