serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the homelab. It protects the Caddy vhosts that do not carry their own application-level authentication with its embedded proxy outpost. Apps that have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped — they handle their own session.

• Version: 2026.8.0 (pinned in dankclaw/authentik/quadlet/*.container)
• Host: dankclaw (192.168.0.143)
• Name: auth.franksdank.cloud
• Stack: 3 compose services — authentik-postgres, authentik-server, authentik-worker (see Pi-hole for the LAN-DNS pattern)

Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in .docaudit/research/auth.md. In short: full OIDC with single-logout, a web GUI for all configuration, an outpost that terminates auth for reverse-proxied apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10, so the stack is 3 services, not 4. (The integration mechanism changed in 2026.x — see Architecture; the research file predates the pin.)

Architecture

graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]

Two mechanisms were disproved live (2026-08-29/30) and one is the actual contract:

• The old /outpost.goauthentik.io/auth/caddy forward-auth endpoint does not exist in the 2026.8 URL tree — every unauthenticated subrequest got authentik's own 404 page.
• X-authentik-meta-outpost/-provider/-app headers are IGNORED for routing in this build: a request carrying Caddy's full meta-header set still 404s, while the same request with only a matching Host answers. The embedded outpost routes solely on the original Host, matched against each attached proxy provider's external_host.
• Consequence: every guarded host needs its own proxy provider (exact external_host), and :8016 is a single uniform reverse_proxy to authentik-server:9000 that preserves Host. Unauthenticated → login flow served in place; authenticated → forwarded to the provider's internal_host (http://caddy:8017), which routes straight to the app. The chain terminates there — nothing dials back into Authentik from :8017.

The three compose services

Service	Role	Image
authentik-postgres	PostgreSQL 16 (users, flows, sessions, events)	docker.io/library/postgres:16-alpine
authentik-server	Core + admin API + the embedded Caddy outpost (port 9000)	ghcr.io/goauthentik/server:2026.8.0
authentik-worker	Background tasks (event processing, provider sync, flows)	ghcr.io/goauthentik/server:2026.8.0

• Postgres is the only stateful store. Sessions, caching, and WebSockets moved to PostgreSQL in 2025.10 (Redis removed).
• Server and worker run the same image; the command selects the role (server / worker).
• The worker runs as 0:0 (rootless: container root maps to the host user that owns the data dir).
• Data persists under /home/dank/authentik/ (bind mounts, repo policy — no named volumes).

Caddy integration

Guarded vhosts reach Caddy's internal listener :8016 (post-Anubis), which proxies every request whole to the Authentik server with Host preserved — there is no forward_auth and no per-vhost block:

http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}

The embedded outpost inside the server process decides by Host: no session → the provider's authentication_flow is served in place; session valid → the request is forwarded to the provider's internal_host (http://caddy:8017), where per-vhost handle blocks reach the apps.

Lockstep rule: the Anubis guard set, the Caddy TLS edge, the :8017 routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients.

Live provider inventory (provisioned via ORM, 2026-08-30), all attached to outpost 7e39655b-… ("authentik Embedded Outpost"), all with internal_host = http://caddy:8017 and all three flows set:

Provider pk	name	external_host	app slug
1	embedded-outpost	https://www.franksdank.cloud	caddy
2	search-proxy	https://search.franksdank.cloud	search
3	hermes-proxy	https://hermes.franksdank.cloud	hermes
4	prometheus-proxy	https://prometheus.franksdank.cloud	prometheus
5	romm-proxy	https://romm.franksdank.cloud	romm
6	files-proxy	https://files.franksdank.cloud	files

Provisioning pitfall (verified live): a ProxyProvider created with only authentication_flow set — authorization_flow/invalidation_flow null — makes the embedded outpost thread fail its own config fetch: the Rust side logs Failed to fetch outpost from API, retrying in 3 seconds forever and the entire /outpost.goauthentik.io space 404s for every host. Backfill all three flows (pk1's values: default-authentication-flow, default-provider-authorization-implicit-consent, default-provider-invalidation-flow) and the retry loop recovers on its next 3-second cycle.

OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (grafana, pk 9, confidential client) was created and verified end-to-end via ak shell ORM drives. This fork's OAuth2 provider implementation diverges from upstream in ways that bit us; all verified live:

• Application row required. OAuth2Provider.objects.create() does not create the single-table-inheritance Application row; without it the flow executor can never complete an authorize request. Create both rows.
• grant_types defaults to empty. Until the provider's grant_types contains "authorization_code" (enum value with underscore), the token endpoint rejects every exchange with invalid_request.
• Token endpoint is form-encoded + Basic-auth only. A JSON body makes parse_token_request see no grant_type → unsupported_grant_type; POST-body client secrets are not accepted — authenticate the confidential client with Authorization: Basic base64(client_id:client_secret).
• Provider resolution is by client_id query param on the global /application/o/authorize/ route; per-slug authorize paths do not exist in this build. Token/userinfo live at /application/o/token/ and /application/o/userinfo/?api=1.
• Property-mapping evaluator is broken → userinfo returns sub only. PropertyMappingEvaluator.evaluate(*args, **kwargs) forwards kwargs to BaseEvaluator.evaluate(expression_source), so the first context kwarg becomes the expression source instead of the context; every call raises a swallowed PropertyMappingExpressionException and the claim is skipped. Scope-grant filtering has the same class of bug: ScopeMapping has no provider FK (linkage lives in the ProviderPropertyMapping through model), so filter(provider=...) always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with {sub} only — email/groups/entitlements claims cannot be delivered by this build until the evaluator is patched or upgraded. Clients consuming this provider must key identity off sub.
• Scope mappings for this provider exist (grafana-email-scope, grafana-profile-scope, Python-syntax expressions) and will start emitting claims the moment the evaluator bug is fixed — no config change needed then.

Configuration (secrets)

Two secrets live in the host .env (git-ignored; placeholders in .env.example):

• AUTHENTIK_PG_PASS — the PostgreSQL password. Generate: openssl rand -base64 32.
• AUTHENTIK_SECRET_KEY — the server's signing key. Generate: openssl rand -base64 48.

First-time setup

1. ./scripts/deploy.sh dankclaw — pulls, starts the 3 services (postgres health-gated), reloads Caddy.
2. Open https://auth.franksdank.cloud — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: Directory → Applications → add an application + a Proxy provider whose external_host is exactly the browser-facing origin (https://<host>), internal_host = http://caddy:8017, and all three flows set (see the provisioning pitfall above). Attach the provider to the embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an OAuth2/OpenID Connect provider, create the app with the correct redirect URI, and paste the client_id / client_secret into the app's SSO settings.

Operations

• Health: https://auth.franksdank.cloud/-/health/ (the server healthcheck probes this).
• Logs: podman logs authentik-server / podman logs authentik-worker.
• Backups: pg_dump the authentik-postgres database (the single source of truth). The /home/dank/authentik/ dir holds media + static files.
• Upgrades: bump the image tag in dankclaw/compose.yaml, run ./scripts/deploy.sh dankclaw. Authentik migrates the schema on startup.

Failure modes

Symptom	Cause	Fix
Every vhost 502s at the outpost	authentik-server down	podman logs authentik-server; check postgres health
Login loops (302 → login → 302)	trusted_proxies / X-Forwarded-* mismatch	Confirm Caddy sets X-Forwarded-For; the outpost trusts private_ranges
"Invalid signature" on the outpost	AUTHENTIK_SECRET_KEY differs between server and worker	Both services read the same .env var; re-deploy
Postgres "password authentication failed"	AUTHENTIK_PG_PASS changed after first init	Reset via pg_ctl or re-init the data dir
"Request has been denied: Invalid next URL" at sign-in	Flow entered with an absolute cross-host next= (old redirect design)	Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie
/outpost.goauthentik.io/* 404s on every host	A provider row with null authorization_flow/invalidation_flow wedges the outpost config fetch	Backfill the flows; watch podman logs authentik-server for Failed to fetch outpost from API
Off-LAN clients get "hostname could not be found"	Guarded host missing from the public DNS zone	Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer

Cross-references

• Caddy — the reverse proxy carrying the :8016/:8017 auth-chain listeners
• Pi-hole — the LAN DNS that resolves auth.franksdank.cloud
• Network Architecture — the flat naming scheme
• Research: .docaudit/research/auth.md — the full provider comparison