Frank
afe72781bc
The vault claimed the fork property-mapping evaluator was broken and userinfo returned only sub, so email could never reach Grafana (outcome unverified). That was a misdiagnosis: the real causes were the NULL authorization_flow (the NoneType.slug crash) and the email scope mapping returning a bare string rather than a dict. Record the four verified causes and fixes, point to the repair script and the new landing spec, and drop the unverified caveat.
102 lines
6.5 KiB
Markdown
102 lines
6.5 KiB
Markdown
---
|
|
type: reference
|
|
description: "Grafana visualizes the metrics that Prometheus collects from every service."
|
|
tags:
|
|
- infrastructure
|
|
- grafana
|
|
timestamp: 2026-10-20
|
|
---
|
|
|
|
# Grafana
|
|
|
|
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).
|
|
|
|
**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/grafana/grafana:latest` |
|
|
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
|
|
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
|
|
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
|
|
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
|
|
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |
|
|
|
|
## Configuration
|
|
|
|
Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is
|
|
the only credential, and systemd expands it from a per-service env file:
|
|
|
|
```
|
|
[Service]
|
|
EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only
|
|
[Container]
|
|
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
|
|
```
|
|
|
|
`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.
|
|
|
|
## Datasource
|
|
|
|
The Prometheus datasource is declared in git, not clicked into `grafana.db`:
|
|
`shared/grafana/provisioning/datasources/prometheus.yml` points at
|
|
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
|
|
address Caddy proxies to, never the loopback-published port — and carries
|
|
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
|
|
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
|
|
alongside it by `dashboards/provider.yml` (provider `fleet`
|
|
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
|
|
|
|
A Loki datasource is still **not** provisioned — the tree contains exactly one
|
|
datasource file — so nothing in Grafana queries logs today, which matters less
|
|
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
|
|
Verification is through the API, not the filesystem:
|
|
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.
|
|
|
|
## Healthcheck
|
|
|
|
Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).
|
|
|
|
## Gotchas
|
|
|
|
- Change the default admin credentials at first boot (env or UI).
|
|
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
|
|
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
|
|
no provider token (`token=None`), so no identity header reaches Grafana; the
|
|
login is Grafana's own "Sign in with Authentik" button against the `grafana`
|
|
confidential-client provider. **Verified end-to-end 2026-10-20** after four
|
|
independent breakages had made it fail. The two Authentik-side causes (a NULL
|
|
`authorization_flow` → authorize 500; an email scope mapping that returned a
|
|
bare string so no `email` claim was emitted) and the full wire contract live in
|
|
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
|
|
re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
|
|
an ak-shell re-creation of the provider or a blueprint import, both of which
|
|
revert it silently). The Grafana half is the unit's own env (`grafana.container`):
|
|
`SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
|
|
authorize view), `USE_ID_TOKEN=true` + no `?api=1` (Grafana v13 otherwise GETs
|
|
`<api_url>/emails`, which authentik 404s and the login dies with "Login failed:
|
|
InternalError"), attribute paths `email`/`sub`/`preferred_username`, and
|
|
`ALLOW_SIGN_UP=true` — safe because the guard fronts every request, so the
|
|
callback is unreachable to unauthenticated traffic.
|
|
- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
|
|
`ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
|
|
`email`. The local admin account stays the only Admin. The landing that the
|
|
edge-only 13-sso spec never checked is now asserted by
|
|
`tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
|
|
an email).
|
|
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
|
|
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
|
|
for the service, podman then looks for its configuration under `/root`
|
|
cannot read it, and the unit dies at start with
|
|
`stat /root/.config: permission denied`. The unit reads
|
|
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
|
|
one variable.
|
|
|
|
## See Also
|
|
|
|
- [[Dankclaw Stack]]
|
|
- [[Prometheus]] — the datasource
|
|
- [[Observability]] — the whole stack
|