serverAppConfigs/serverAppConfigs-docs/Infrastructure/Grafana.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Grafana visualizes the metrics that Prometheus collects from every service.	infrastructure grafana	2026-09-11

Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at grafana.franksdank.cloud (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

Host: Dankclaw — Hostname: grafana.franksdank.cloud — Port: 3002 (internal) — Docs: https://grafana.com/docs

Facts

Item	Value
Image	docker.io/grafana/grafana:latest
Published	127.0.0.1:3002 → 3000, loopback only — never a wildcard (dankclaw/observability/quadlet/grafana.container:40; ss -ltnp 2026-10-18 shows 127.0.0.1:3002 and no other bind)
Unit	dankclaw/observability/quadlet/grafana.container → grafana.service; Restart=always/RestartSec=10 + WantedBy=default.target (grafana.container:75-79), installed at ~/.config/containers/systemd/grafana.container, reporting generated + active (`systemctl --user is-enabled
Data	/opt/grafana/data — grafana.db (present on the host: ls /opt/grafana/data/grafana.db, 2026-10-18): accounts, sessions, anything created in the UI
Provisioning mount	/opt/grafana/provisioning (read-only, grafana.container:57) — populated as code: datasources/prometheus.yml, dashboards/provider.yml, dashboards/llm-fleet.json are on the host (ls -la /opt/grafana/provisioning/* 2026-10-18) and are installed on every deploy by copy_grafana_provisioning (scripts/deploy-remote.sh:286-300). History: the mount was empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit 6a19143); tests/observability_quadlet.bats:345 pins the tree
Datasource	one, provisioned: Prometheus, uid prometheus, http://prometheus:9090, default, not editable, timeInterval: 15s (shared/grafana/provisioning/datasources/prometheus.yml:14-26). No Loki datasource exists — see Loki

Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is the only credential, and systemd expands it from a per-service env file:

[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}

seed_grafana_env in scripts/deploy-remote.sh:477-486 seeds that file from dankclaw/.env with a single-variable filter (awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …', mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

Datasource

The Prometheus datasource is declared in git, not clicked into grafana.db: shared/grafana/provisioning/datasources/prometheus.yml points at http://prometheus:9090 — the container name on dankclaw_default, the same address Caddy proxies to, never the loopback-published port — and carries timeInterval: 15s so Grafana's rate windows cannot be shorter than the scrape interval (prometheus.yml:5). The llm-fleet dashboard is provisioned alongside it by dashboards/provider.yml (provider fleet, updateIntervalSeconds: 30, allowUiUpdates: true).

A Loki datasource is still not provisioned — the tree contains exactly one datasource file — so nothing in Grafana queries logs today, which matters less than it sounds because Loki holds no streams (see Loki → Contents). Verification is through the API, not the filesystem: curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources.

Healthcheck

Grafana responds on /api/health with 200 and "database":"ok"; measured 2026-10-18 on http://127.0.0.1:3002/api/health, which also reports version 13.2.2 (podman ps reports the container healthy on the unit's wget --spider -q http://localhost:3000/api/health, grafana.container:58).

Gotchas

• Change the default admin credentials at first boot (env or UI).
• Dashboards provisioned as code (dashboard JSON plus a YAML provider, shared/grafana/provisioning/dashboards/) survive container recreation; UI-created ones live in the data volume — back it up.
• Auth = OAuth2 relying-party, not auth_proxy. The embedded outpost carries no provider token (token=None), so no identity header ever reaches Grafana; RP mode is topology-independent. The full verified contract (provider pk 9, client_id grafana-21ca190cde75d60f, global authorize/token/userinfo paths, Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the sub hash only) is in Authentik → "OAuth2 relying-party quirks". Consequences for Grafana: The unit sets ALLOW_SIGN_UP=false, AUTO_UPSERT=true, ALLOW_ASSIGN_ORG=true, ORG_ID=1 and no role-attribute mapping (grafana.container:52-55), so users land as Viewer with login = <sub hex>; no email/groups/role claims reach Grafana until the evaluator bug is fixed — do not write role mappings that depend on those claims. The local admin account stays the only Admin. (The Viewer-default outcome and the <sub hex> login are not readable from the host without the database or an admin password. (unverified 2026-10-18))
• Never point this unit's EnvironmentFile at the repo-root .env. That file is a gluetun env file and it carries HOME=/root; systemd imports it for the service, podman then looks for its configuration under /root, cannot read it, and the unit dies at start with stat /root/.config: permission denied. The unit reads /home/dank/Containers/grafana/env, which the deployer seeds with exactly one variable.

See Also

• Dankclaw Stack
• Prometheus — the datasource
• Observability — the whole stack