serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the homelab. It protects the Caddy vhosts that do not carry their own application-level authentication with its embedded proxy outpost. Apps that have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped — they handle their own session.

• Version: 2026.8.0 (pinned in dankclaw/authentik/quadlet/*.container)
• Host: dankclaw (192.168.0.143)
• Name: auth.franksdank.cloud
• Stack: 3 compose services — authentik-postgres, authentik-server, authentik-worker (see Pi-hole for the LAN-DNS pattern)
• LDAP outpost: a 4th, quadlet-deployed unit (authentik-outpost-ldap) serves LDAP for Jellyfin — see the section below.

Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in .docaudit/research/auth.md. In short: full OIDC with single-logout, a web GUI for all configuration, an outpost that terminates auth for reverse-proxied apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10, so the stack is 3 services, not 4. (The integration mechanism changed in 2026.x — see Architecture; the research file predates the pin.)

Architecture

graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]

Two mechanisms were disproved live (2026-08-29/30) and one is the actual contract:

• The old /outpost.goauthentik.io/auth/caddy forward-auth endpoint does not exist in the 2026.8 URL tree — every unauthenticated subrequest got authentik's own 404 page.
• X-authentik-meta-outpost/-provider/-app headers are IGNORED for routing in this build: a request carrying Caddy's full meta-header set still 404s, while the same request with only a matching Host answers. The embedded outpost routes solely on the original Host, matched against each attached proxy provider's external_host.
• Consequence: every guarded host needs its own proxy provider (exact external_host), and :8016 is a single uniform reverse_proxy to authentik-server:9000 that preserves Host. Unauthenticated → login flow served in place; authenticated → forwarded to the provider's internal_host (http://caddy:8017), which routes straight to the app. The chain terminates there — nothing dials back into Authentik from :8017.

The three compose services

Service	Role	Image
authentik-postgres	PostgreSQL 16 (users, flows, sessions, events)	docker.io/library/postgres:16-alpine
authentik-server	Core + admin API + the embedded Caddy outpost (port 9000)	ghcr.io/goauthentik/server:2026.8.0
authentik-worker	Background tasks (event processing, provider sync, flows)	ghcr.io/goauthentik/server:2026.8.0

• Postgres is the only stateful store. Sessions, caching, and WebSockets moved to PostgreSQL in 2025.10 (Redis removed).
• Server and worker run the same image; the command selects the role (server / worker).
• The worker runs as 0:0 (rootless: container root maps to the host user that owns the data dir).
• Data persists under /home/dank/Containers/authentik/{data,postgres} (bind mounts, repo policy — no named volumes; moved from /home/dank/authentik/ in the 2026-09-01 quadlet standardization).

Caddy integration

Guarded vhosts reach Caddy's internal listener :8016 (post-Anubis), which proxies every request whole to the Authentik server with Host preserved — there is no forward_auth and no per-vhost block:

http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}

The embedded outpost inside the server process decides by Host: no session → the provider's authentication_flow is served in place; session valid → the request is forwarded to the provider's internal_host (http://caddy:8017), where per-vhost handle blocks reach the apps.

Lockstep rule: the Anubis guard set, the Caddy TLS edge, the :8017 routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients.

Live provider inventory (verified 2026-09-02), all attached to outpost 7e39655b-… ("authentik Embedded Outpost"), all with internal_host = http://caddy:8017, all three flows set, and each linked to an Application (see the provisioning rules below):

Provider pk	name	external_host	app slug
1	embedded-outpost	https://www.franksdank.cloud	caddy
2	search-proxy	https://search.franksdank.cloud	search
4	prometheus-proxy	https://prometheus.franksdank.cloud	prometheus
5	romm-proxy	https://romm.franksdank.cloud	romm
6	files-proxy	https://files.franksdank.cloud	files
7	loki-proxy	https://loki.franksdank.cloud/	loki
8	grafana-proxy	https://grafana.franksdank.cloud/	grafana-guard
10	seerr-proxy	https://seerr.franksdank.cloud/	seerr-guard

Provider 3 (hermes-proxy) still exists but is dormant — its vhost was decommissioned from Caddy on 2026-08-31. Provider 9 (grafana) and 12 (Seerr OIDC) are OAuth2 relying-party providers, not proxy providers; the outpost does not deserialize them. Trailing slashes on external_host (the runbook-recipe style) and bare origins both intercept correctly — the outpost matches on hostname.

Two provisioning rules, each verified live by an outage:

1. All three flows are required. A ProxyProvider created via raw API/ORM with any of authentication_flow/authorization_flow/invalidation_flow null wedges the embedded outpost's config fetch: the Rust side rejects the whole config JSON (serde: invalid type: null, expected a formatted UUID string), logs Failed to fetch outpost from API, retrying in 3 seconds forever, and everything behind the outpost dies at the next outpost reload — including hosts whose providers were fine (verified 2026-09-02 when three providers provisioned without any flows took down every guarded vhost after a server restart). Backfill all three flows (copy from pk 1: default-authentication-flow, default-provider-authorization-implicit-consent (the implicit-consent authorization flow, default-invalidation-flow) and the retry loop recovers on its next 3-second cycle. The silent killer: while the outpost process keeps its last good in-memory config, everything works — the outage only surfaces on restart/reload.
2. Every proxy provider needs an attached Application. The outpost resolves the post-login redirect through the provider's application; a provider with no app completes the login (authentik session established) but bounces to authentik's own user UI (/if/user/#/library) instead of the app. No error is logged anywhere — the SSO "works" and the app never appears. Create the Application with provider_id set (the UI does this automatically; raw API/ORM drives must do it explicitly).

OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (grafana, pk 9, confidential client) was created and verified end-to-end via ak shell ORM drives. This fork's OAuth2 provider implementation diverges from upstream in ways that bit us; all verified live:

• Application row required. OAuth2Provider.objects.create() does not create the single-table-inheritance Application row; without it the flow executor can never complete an authorize request. Create both rows.
• grant_types defaults to empty. Until the provider's grant_types contains "authorization_code" (enum value with underscore), the token endpoint rejects every exchange with invalid_request.
• Token endpoint is form-encoded + Basic-auth only. A JSON body makes parse_token_request see no grant_type → unsupported_grant_type; POST-body client secrets are not accepted — authenticate the confidential client with Authorization: Basic base64(client_id:client_secret).
• Provider resolution is by client_id query param on the global /application/o/authorize/ route; per-slug authorize paths do not exist in this build. Token/userinfo live at /application/o/token/ and /application/o/userinfo/?api=1.
• Property-mapping evaluator is broken → userinfo returns sub only. PropertyMappingEvaluator.evaluate(*args, **kwargs) forwards kwargs to BaseEvaluator.evaluate(expression_source), so the first context kwarg becomes the expression source instead of the context; every call raises a swallowed PropertyMappingExpressionException and the claim is skipped. Scope-grant filtering has the same class of bug: ScopeMapping has no provider FK (linkage lives in the ProviderPropertyMapping through model), so filter(provider=...) always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with {sub} only — email/groups/entitlements claims cannot be delivered by this build until the evaluator is patched or upgraded. Clients consuming this provider must key identity off sub.
• Scope mappings for this provider exist (grafana-email-scope, grafana-profile-scope, Python-syntax expressions) and will start emitting claims the moment the evaluator bug is fixed — no config change needed then.

LDAP outpost (Jellyfin auth, live 2026-09-01)

A fourth unit, dankclaw/authentik/quadlet/authentik-outpost-ldap.container → authentik-outpost-ldap.service, serves LDAP for Jellyfin's LDAP-Auth plugin. The image is pinned by its amd64 single-platform digest (ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4): podman 5.7.0 refuses the ghcr.io/goauthentik/ldap:2026.8.0 tag because the tag resolves to a nested manifest list (index → per-platform manifest lists) — unexpectedly received a manifest list instead of a manifest for a single image. Resolve once with podman manifest inspect <image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest' and pin that digest.

• Ports 389 (ldap) and 636 (ldaps) publish on the LAN; Jellyfin on Dankserver binds 192.168.0.143:389. The outpost token travels in AUTHENTIK_LDAP_OUTPOST_TOKEN from the host .env (git-ignored).

Authentik-side objects (provisioned via ak shell ORM)

Object	Value
LDAP provider	Jellyfin LDAP (pk 11), base DN dc=franksdank,dc=cloud
Bind DN	cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud
Service account	svc-jellyfin-ldap — passwordless/unusable; authenticates by token
Bind token	intent verification, non-expiring — its key is the LDAP bind password
RBAC role	svc-jellyfin-ldap-search carrying object-level permission authentik_providers_ldap.search_full_directory on the provider
Groups	jellyfin_users, jellyfin_admins
Application	jellyfin (slug jellyfin)

Hard-won rules, each verified live:

• Token intent is load-bearing. A bind token with intent=app_password is rejected by the flow's password stage with Invalid password; intent=verification passes. The outpost's direct binder executes the provider's authorization flow, and that password stage only accepts verification-intent tokens as static passwords.
• Search needs search_full_directory. Without the RBAC grant the outpost returns only the bind user itself — a (memberOf=…) filter then matches zero entries and the plugin logs Found no users matching X. The permission is object-level on the provider; grant it via an RBAC role and retest.
• ak shell import path for the provider model in 2026.8: from authentik.providers.ldap.models import LDAPProvider.

Configuration (secrets)

Two secrets live in the host .env (git-ignored; placeholders in .env.example):

• AUTHENTIK_PG_PASS — the PostgreSQL password. Generate: openssl rand -base64 32.
• AUTHENTIK_SECRET_KEY — the server's signing key. Generate: openssl rand -base64 48.

First-time setup

1. ./scripts/deploy.sh dankclaw — pulls, starts the 3 services (postgres health-gated), reloads Caddy.
2. Open https://auth.franksdank.cloud — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: Directory → Applications → add an application + a Proxy provider whose external_host is exactly the browser-facing origin (https://<host>), internal_host = http://caddy:8017, and all three flows set (see the provisioning pitfall above). Attach the provider to the embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an OAuth2/OpenID Connect provider, create the app with the correct redirect URI, and paste the client_id / client_secret into the app's SSO settings.

Operations

• Health: https://auth.franksdank.cloud/-/health/ (the server healthcheck probes this).
• Logs: podman logs authentik-server / podman logs authentik-worker.
• Backups: pg_dump the authentik-postgres database (the single source of truth). The /home/dank/Containers/authentik/data dir holds media + static files.
• Upgrades: bump the image tag in the three units under dankclaw/authentik/quadlet/, run ./scripts/deploy.sh dankclaw. Authentik migrates the schema on startup.

Outpost claim cache (proxysession) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to authentik_providers_proxy_proxysession whose session_data JSON holds the claims (groups included) that get served for that browser session until the row expires (observed TTLs 1–24 h per row). A session authorized before a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';

The silent re-auth path was proven end-to-end: after deleting qa-e2e-admin's rows, the e2e admin spec ran against the stale storage state and passed with no login prompt (2 tests, 2.9 s).

Failure modes

Symptom	Cause	Fix
Every vhost 502s at the outpost	authentik-server down	podman logs authentik-server; check postgres health
Login loops (302 → login → 302)	trusted_proxies / X-Forwarded-* mismatch	Confirm Caddy sets X-Forwarded-For; the outpost trusts private_ranges
"Invalid signature" on the outpost	AUTHENTIK_SECRET_KEY differs between server and worker	Both services read the same .env var; re-deploy
Postgres "password authentication failed"	AUTHENTIK_PG_PASS changed after first init	Reset via pg_ctl or re-init the data dir
"Request has been denied: Invalid next URL" at sign-in	Flow entered with an absolute cross-host next= (old redirect design)	Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie
/outpost.goauthentik.io/* 404s on every host	Any provider row with a null flow (authentication/authorization/invalidation) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload	Backfill all three flows on every provider; watch podman logs authentik-server for Failed to fetch outpost from API
Login completes but lands on authentik's user UI (/if/user/…) instead of the app	Proxy provider has no attached Application (outpost cannot resolve the post-login redirect)	Create the Application with provider_id set — no error is logged, so test with a fresh browser context
Off-LAN clients get "hostname could not be found"	Guarded host missing from the public DNS zone	Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer
Correct membership but one user still 403s at a header gate	Outpost claim cache: a proxysession row minted before the membership change serves old claims until expiry	Delete that user's proxysession rows (section above) — next visit re-authorizes silently

Cross-references

• Caddy — the reverse proxy carrying the :8016/:8017 auth-chain listeners
• Pi-hole — the LAN DNS that resolves auth.franksdank.cloud
• Jellyfin — consumes the LDAP outpost via its LDAP-Auth plugin
• Network Architecture — the flat naming scheme
• Research: .docaudit/research/auth.md — the full provider comparison