The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.
Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.
scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.
grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
Frank