---
name: grafana-provisioning-as-code
description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
---

# Grafana provisioning as code (Dankclaw fleet)

Use when adding or changing a Grafana datasource or dashboard on this fleet, or
when a dashboard "does not appear" after a deploy.

## Shape of the thing

Grafana on Dankclaw is an **empty install configured entirely from git**. The
quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
provisioned artefact exists in exactly three places:

```
shared/grafana/provisioning/datasources/<name>.yml    # repo: source of truth
shared/grafana/provisioning/dashboards/provider.yml   # dashboard loader config
shared/grafana/provisioning/dashboards/<name>.json    # dashboards
        |  deploy-remote.sh: copy_grafana_provisioning()
        v
/opt/grafana/provisioning/...                          # host: installed state
        |  read-only mount
        v
/etc/grafana/provisioning/...                          # generated: inside container
```

Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
invisible to review, unknown to anyone reading the repo, and lost with the data
volume.

## Procedure

1. Write the files under `shared/grafana/provisioning/`.
   - Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
     uid, so a random one breaks them), `access: proxy`, and the URL by
     **container name** (`http://prometheus:9090`) — the published port is
     loopback-only and unreachable from inside the network.
   - Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
     failing an operator who wants a different rate window is the wrong trade)
     and a JSON dashboard under a stable `uid`.
2. Add the file(s) to `copy_grafana_provisioning()` in
   `scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
   file — follow `copy_prometheus_config` for style).
3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
   with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
   the datasource URL must be the container name, the unit must mount
   `/opt/grafana/provisioning`, and the deployer must call the copier. Then
   `./tests/run.sh`.
5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
   provisioning tree, restarts Grafana, then runs the e2e gate).

## Verify — and the trap that makes the obvious check lie

A **provisioned dashboard does not appear in the legacy `dashboard` table** of
`grafana.db` on the current Grafana build; dashboards live in unified storage
behind the `dashboard.grafana.app` API group. The datasource *does* land in
`data_source`, and that asymmetry is easy to misread as "datasource worked,
dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
negative**, not evidence.

Check in this order instead:

```bash
# 1. provisioning actually ran (these loggers name themselves)
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"

# 2. the dashboard exists (uid is what you set in the JSON)
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"

# 3. the datasource really reaches Prometheus (proves the whole chain)
curl -s -u admin:admin \
  "http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
```

Grafana publishes on loopback `:3002` only; the off-host path is the guarded
vhost. Its admin login is still the default `admin:admin` — which is why every
assertion above needs credentials, and which is worth changing.

## Before writing any panel or config comment

Grep the metric names out of the live endpoint; naming a series that does not
exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
panel that silently renders "No data", is worse than no panel:

```bash
curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
```