Frank
d8196ce499
The login lands once authentik emits the email claim (the IdP fix); Grafana needs no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in the literal-secret seam. Correct the two SSO notes accordingly.
102 lines
6.5 KiB
Markdown
102 lines
6.5 KiB
Markdown
---
|
|
type: reference
|
|
description: "Grafana visualizes the metrics that Prometheus collects from every service."
|
|
tags:
|
|
- infrastructure
|
|
- grafana
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Grafana
|
|
|
|
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).
|
|
|
|
**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/grafana/grafana:latest` |
|
|
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
|
|
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
|
|
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
|
|
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
|
|
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |
|
|
|
|
## Configuration
|
|
|
|
Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is
|
|
the only credential, and systemd expands it from a per-service env file:
|
|
|
|
```
|
|
[Service]
|
|
EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only
|
|
[Container]
|
|
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
|
|
```
|
|
|
|
`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.
|
|
|
|
## Datasource
|
|
|
|
The Prometheus datasource is declared in git, not clicked into `grafana.db`:
|
|
`shared/grafana/provisioning/datasources/prometheus.yml` points at
|
|
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
|
|
address Caddy proxies to, never the loopback-published port — and carries
|
|
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
|
|
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
|
|
alongside it by `dashboards/provider.yml` (provider `fleet`
|
|
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
|
|
|
|
A Loki datasource is still **not** provisioned — the tree contains exactly one
|
|
datasource file — so nothing in Grafana queries logs today, which matters less
|
|
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
|
|
Verification is through the API, not the filesystem:
|
|
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.
|
|
|
|
## Healthcheck
|
|
|
|
Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).
|
|
|
|
## Gotchas
|
|
|
|
- Change the default admin credentials at first boot (env or UI).
|
|
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
|
|
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
|
|
no provider token (`token=None`), so no identity header reaches Grafana; the
|
|
login is Grafana's own "Sign in with Authentik" button against the `grafana`
|
|
confidential-client provider. **Verified end-to-end 2026-10-20** after four
|
|
independent breakages had made it fail. The two Authentik-side causes (a NULL
|
|
`authorization_flow` → authorize 500; an email scope mapping that returned a
|
|
bare string so no `email` claim was emitted) and the full wire contract live in
|
|
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
|
|
re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
|
|
an ak-shell re-creation of the provider or a blueprint import, both of which
|
|
revert it silently). The Grafana half is the unit's own env (`grafana.container`):
|
|
`SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
|
|
authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
|
|
attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
|
|
(safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
|
|
when the token carries no `email` claim — so it is the email claim itself (an
|
|
IdP fix, above) that makes the login land, not any Grafana toggle.
|
|
- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
|
|
`ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
|
|
`email`. The local admin account stays the only Admin. The landing that the
|
|
edge-only 13-sso spec never checked is now asserted by
|
|
`tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
|
|
an email).
|
|
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
|
|
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
|
|
for the service, podman then looks for its configuration under `/root`
|
|
cannot read it, and the unit dies at start with
|
|
`stat /root/.config: permission denied`. The unit reads
|
|
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
|
|
one variable.
|
|
|
|
## See Also
|
|
|
|
- [[Dankclaw Stack]]
|
|
- [[Prometheus]] — the datasource
|
|
- [[Observability]] — the whole stack
|