---
type: reference
description: "Grafana visualizes the metrics that Prometheus collects from every service."
tags:
  - infrastructure
  - grafana
timestamp: 2026-10-20
---

# Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs

## Facts

| Item | Value |
|------|-------|
| Image | `docker.io/grafana/grafana:latest` |
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |

## Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is
the only credential, and systemd expands it from a per-service env file:

```
[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
```

`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

## Datasource

The Prometheus datasource is declared in git, not clicked into `grafana.db`:
`shared/grafana/provisioning/datasources/prometheus.yml` points at
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).

A Loki datasource is still **not** provisioned — the tree contains exactly one
datasource file — so nothing in Grafana queries logs today, which matters less
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
Verification is through the API, not the filesystem:
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.

## Healthcheck

Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).

## Gotchas

- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
  no provider token (`token=None`), so no identity header reaches Grafana; the
  login is Grafana's own "Sign in with Authentik" button against the `grafana`
  confidential-client provider. **Verified end-to-end 2026-10-20** after four
  independent breakages had made it fail. The two Authentik-side causes (a NULL
  `authorization_flow` → authorize 500; an email scope mapping that returned a
  bare string so no `email` claim was emitted) and the full wire contract live in
  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
  an ak-shell re-creation of the provider or a blueprint import, both of which
  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
  when the token carries no `email` claim — so it is the email claim itself (an
  IdP fix, above) that makes the login land, not any Grafana toggle.
- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
  `email`. The local admin account stays the only Admin. The landing that the
  edge-only 13-sso spec never checked is now asserted by
  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
  an email).
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`
  cannot read it, and the unit dies at start with
  `stat /root/.config: permission denied`. The unit reads
  `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
  one variable.

## See Also

- [[Dankclaw Stack]]
- [[Prometheus]] — the datasource
- [[Observability]] — the whole stack