Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
93 lines
5.8 KiB
Markdown
93 lines
5.8 KiB
Markdown
---
|
|
type: reference
|
|
description: "Grafana visualizes the metrics that Prometheus collects from every service."
|
|
tags:
|
|
- infrastructure
|
|
- grafana
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Grafana
|
|
|
|
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).
|
|
|
|
**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `docker.io/grafana/grafana:latest` |
|
|
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
|
|
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
|
|
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
|
|
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
|
|
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |
|
|
|
|
## Configuration
|
|
|
|
Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is
|
|
the only credential, and systemd expands it from a per-service env file:
|
|
|
|
```
|
|
[Service]
|
|
EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only
|
|
[Container]
|
|
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
|
|
```
|
|
|
|
`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.
|
|
|
|
## Datasource
|
|
|
|
The Prometheus datasource is declared in git, not clicked into `grafana.db`:
|
|
`shared/grafana/provisioning/datasources/prometheus.yml` points at
|
|
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
|
|
address Caddy proxies to, never the loopback-published port — and carries
|
|
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
|
|
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
|
|
alongside it by `dashboards/provider.yml` (provider `fleet`,
|
|
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
|
|
|
|
A Loki datasource is still **not** provisioned — the tree contains exactly one
|
|
datasource file — so nothing in Grafana queries logs today, which matters less
|
|
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
|
|
Verification is through the API, not the filesystem:
|
|
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.
|
|
|
|
## Healthcheck
|
|
|
|
Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).
|
|
|
|
## Gotchas
|
|
|
|
- Change the default admin credentials at first boot (env or UI).
|
|
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
|
|
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
|
|
no provider token (`token=None`), so no identity header ever reaches Grafana;
|
|
RP mode is topology-independent. The full verified contract (provider pk 9,
|
|
client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
|
|
Basic client auth, form-encoded token body, and the fork's userinfo-claims
|
|
bug that leaves identity at the `sub` hash only) is in
|
|
[[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
|
|
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
|
|
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
|
|
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
|
|
reach Grafana until the evaluator bug is fixed — do not write role mappings
|
|
that depend on those claims. The local admin account stays the only Admin.
|
|
(The Viewer-default outcome and the `<sub hex>` login are not readable from
|
|
the host without the database or an admin password. *(unverified 2026-10-18)*)
|
|
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
|
|
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
|
|
for the service, podman then looks for its configuration under `/root`,
|
|
cannot read it, and the unit dies at start with
|
|
`stat /root/.config: permission denied`. The unit reads
|
|
`/home/dank/Containers/grafana/env`, which the deployer seeds with exactly
|
|
one variable.
|
|
|
|
## See Also
|
|
|
|
- [[Dankclaw Stack]]
|
|
- [[Prometheus]] — the datasource
|
|
- [[Observability]] — the whole stack
|