---
type: reference
description: "Grafana visualizes the metrics that Prometheus collects from every service."
tags:
  - infrastructure
  - grafana
timestamp: 2026-09-11
---

# Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs

## Facts

| Item | Value |
|------|-------|
| Image | `docker.io/grafana/grafana:latest` |
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |

## Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is
the only credential, and systemd expands it from a per-service env file:

```
[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
```

`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

## Datasource

The Prometheus datasource is declared in git, not clicked into `grafana.db`:
`shared/grafana/provisioning/datasources/prometheus.yml` points at
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).

A Loki datasource is still **not** provisioned — the tree contains exactly one
datasource file — so nothing in Grafana queries logs today, which matters less
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
Verification is through the API, not the filesystem:
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.

## Healthcheck

Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).

## Gotchas

- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
  no provider token (`token=None`), so no identity header ever reaches Grafana;
  RP mode is topology-independent. The full verified contract (provider pk 9,
  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
  Basic client auth, form-encoded token body, and the fork's userinfo-claims
  bug that leaves identity at the `sub` hash only) is in
  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
  reach Grafana until the evaluator bug is fixed — do not write role mappings
  that depend on those claims. The local admin account stays the only Admin.
  (The Viewer-default outcome and the `<sub hex>` login are not readable from
  the host without the database or an admin password. *(unverified 2026-10-18)*)
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`
  cannot read it, and the unit dies at start with
  `stat /root/.config: permission denied`. The unit reads
  `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
  one variable.

## See Also

- [[Dankclaw Stack]]
- [[Prometheus]] — the datasource
- [[Observability]] — the whole stack