Files

T

Frank d8196ce499 docs: Grafana reads claims from userinfo, not USE_ID_TOKEN

The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.

2026-10-20 20:41:57 -07:00

6.5 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at grafana.franksdank.cloud (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

Host: Dankclaw — Hostname: grafana.franksdank.cloud — Port: 3002 (internal) — Docs: https://grafana.com/docs

Facts

Item	Value
Image	`docker.io/grafana/grafana:latest`
Published	`127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind)
Unit	`dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Data	`/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI
Provisioning mount	`/opt/grafana/provisioning` (read-only, `grafana.container:57`) — populated as code: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount was* empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree
Datasource	one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see Loki

Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is the only credential, and systemd expands it from a per-service env file:

[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}

seed_grafana_env in scripts/deploy-remote.sh:477-486 seeds that file from dankclaw/.env with a single-variable filter (awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …', mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

Datasource

The Prometheus datasource is declared in git, not clicked into grafana.db: shared/grafana/provisioning/datasources/prometheus.yml points at http://prometheus:9090 — the container name on dankclaw_default, the same address Caddy proxies to, never the loopback-published port — and carries timeInterval: 15s so Grafana's rate windows cannot be shorter than the scrape interval (prometheus.yml:5). The llm-fleet dashboard is provisioned alongside it by dashboards/provider.yml (provider fleet updateIntervalSeconds: 30, allowUiUpdates: true).

A Loki datasource is still not provisioned — the tree contains exactly one datasource file — so nothing in Grafana queries logs today, which matters less than it sounds because Loki holds no streams (see Loki → Contents). Verification is through the API, not the filesystem: curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources.

Healthcheck

Grafana responds on /api/health with 200 and "database":"ok"; measured 2026-10-18 on http://127.0.0.1:3002/api/health, which also reports version 13.2.2 (podman ps reports the container healthy on the unit's wget --spider -q http://localhost:3000/api/health, grafana.container:58).

Gotchas

Change the default admin credentials at first boot (env or UI).
Dashboards provisioned as code (dashboard JSON plus a YAML provider, shared/grafana/provisioning/dashboards/) survive container recreation; UI-created ones live in the data volume — back it up.
Auth = OAuth2 relying-party, not auth_proxy. The embedded outpost carries no provider token (token=None), so no identity header reaches Grafana; the login is Grafana's own "Sign in with Authentik" button against the grafana confidential-client provider. Verified end-to-end 2026-10-20 after four independent breakages had made it fail. The two Authentik-side causes (a NULL authorization_flow → authorize 500; an email scope mapping that returned a bare string so no email claim was emitted) and the full wire contract live in Authentik Outpost Reference → "OAuth2 relying-party quirks"; the IdP half is re-applied by scripts/authentik-grafana-sso-fix.sh (idempotent — re-run after an ak-shell re-creation of the provider or a blueprint import, both of which revert it silently). The Grafana half is the unit's own env (grafana.container): SCOPES="openid email profile" (a GitLab-style user:email scope crashes the authorize view), the standard userinfo API_URL with no ?api=1, the attribute paths email/sub/preferred_username, and ALLOW_SIGN_UP=true (safe: the guard fronts the callback). Grafana v13 only GETs <api_url>/emails when the token carries no email claim — so it is the email claim itself (an IdP fix, above) that makes the login land, not any Grafana toggle.
What a new login becomes: a Viewer in org 1 (ALLOW_ASSIGN_ORG=true, ORG_ID=1, no role-attribute mapping), with login = <sub hex> but the real email. The local admin account stays the only Admin. The landing that the edge-only 13-sso spec never checked is now asserted by tests/e2e/specs/20-grafana-sso.spec.ts (RP round-trip → /api/user 200 with an email).
Never point this unit's EnvironmentFile at the repo-root .env. That file is a gluetun env file and it carries HOME=/root; systemd imports it for the service, podman then looks for its configuration under /root cannot read it, and the unit dies at start with stat /root/.config: permission denied. The unit reads /home/dank/Containers/grafana/env, which the deployer seeds with exactly one variable.

6.5 KiB Raw Blame History