Files

T

Frank 701ab14f60 docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.

2026-09-11 16:10:27 -07:00

3.3 KiB

Raw Blame History

Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at grafana.franksdank.cloud (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

Host: Dankclaw — Hostname: grafana.franksdank.cloud — Port: 3002 (internal) — Docs: https://grafana.com/docs

Facts

Item	Value
Image	`docker.io/grafana/grafana:latest`
Published	`127.0.0.1:3002` → 3000, loopback only (see Observability → Exposure)
Unit	`dankclaw/observability/quadlet/grafana.container` → `grafana.service`
Data	`/opt/grafana/data` — `grafana.db`: accounts, sessions, anything created in the UI
Provisioning mount	`/opt/grafana/provisioning` (read-only) — empty as of 2026-09-11
Datasource	none configured — see the state note in Observability

Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is the only credential, and systemd expands it from a per-service env file:

[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}

seed_grafana_env in scripts/deploy-remote.sh seeds that file from dankclaw/.env with a single-variable filter and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

Datasource

None is configured. Grafana cannot query anything yet; pointing it at http://prometheus:9090 and http://loki:3100 (by container name, on the dankclaw_default network) is outstanding work — see the state note in Observability.

Healthcheck

Grafana responds on /api/health with 200 and "database":"ok".

Gotchas

Change the default admin credentials at first boot (env or UI).
Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
Auth = OAuth2 relying-party, not auth_proxy. The embedded outpost carries no provider token (token=None), so no identity header ever reaches Grafana; RP mode is topology-independent. The full verified contract (provider pk 9, client_id grafana-21ca190cde75d60f, global authorize/token/userinfo paths, Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the sub hash only) is in Authentik → "OAuth2 relying-party quirks". Consequences for Grafana: users auto-upsert as Viewer with login = <sub hex>; no email/groups/role claims reach Grafana until the evaluator bug is fixed — do not write role mappings that depend on those claims. The local admin account stays the only Admin.
Never point this unit's EnvironmentFile at the repo-root .env. That file is a gluetun env file and it carries HOME=/root; systemd imports it for the service, podman then looks for its configuration under /root, cannot read it, and the unit dies at start with stat /root/.config: permission denied. The unit reads /home/dank/Containers/grafana/env, which the deployer seeds with exactly one variable.

3.3 KiB Raw Blame History