serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the	infrastructure authentik	2026-09-16

Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the homelab. It protects the Caddy vhosts that do not carry their own application-level authentication with its embedded proxy outpost. Apps that have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped — they handle their own session.

• Version: 2026.8.3 (pinned in dankclaw/authentik/quadlet/*.container)
• Host: dankclaw (192.168.0.143)
• Name: auth.franksdank.cloud
• Stack: 3 compose services — authentik-postgres, authentik-server, authentik-worker (see Pi-hole for the LAN-DNS pattern)
• LDAP outpost: a 4th, quadlet-deployed unit (authentik-outpost-ldap) serves LDAP for Jellyfin — see the section below.

Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in .docaudit/research/auth.md. In short: full OIDC with single-logout, a web GUI for all configuration, an outpost that terminates auth for reverse-proxied apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10, so the stack is 3 services, not 4. (The integration mechanism changed in 2026.x — see Architecture; the research file predates the pin.)

Architecture

graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]

Two mechanisms were disproved live (2026-08-29/30) and one is the actual contract:

• The old /outpost.goauthentik.io/auth/caddy forward-auth endpoint does not exist in the 2026.8 URL tree — every unauthenticated subrequest got authentik's own 404 page.
• X-authentik-meta-outpost/-provider/-app headers are IGNORED for routing in this build: a request carrying Caddy's full meta-header set still 404s, while the same request with only a matching Host answers. The embedded outpost routes solely on the original Host, matched against each attached proxy provider's external_host.
• Consequence: every guarded host needs its own proxy provider (exact external_host), and :8016 is a single uniform reverse_proxy to authentik-server:9000 that preserves Host. Unauthenticated → login flow served in place; authenticated → forwarded to the provider's internal_host (http://caddy:8017), which routes straight to the app. The chain terminates there — nothing dials back into Authentik from :8017.

The three compose services

Service	Role	Image
authentik-postgres	PostgreSQL 16 (users, flows, sessions, events)	docker.io/library/postgres:16-alpine
authentik-server	Core + admin API + the embedded Caddy outpost (port 9000)	ghcr.io/goauthentik/server:2026.8.3
authentik-worker	Background tasks (event processing, provider sync, flows)	ghcr.io/goauthentik/server:2026.8.3

• Postgres is the only stateful store. Sessions, caching, and WebSockets moved to PostgreSQL in 2025.10 (Redis removed).
• Server and worker run the same image; the command selects the role (server / worker).
• The worker runs as 0:0 (rootless: container root maps to the host user that owns the data dir).
• Data persists under /home/dank/Containers/authentik/{data,postgres} (bind mounts, repo policy — no named volumes; moved from /home/dank/authentik/ in the 2026-09-01 quadlet standardization).

Caddy integration

Guarded vhosts reach Caddy's internal listener :8016 (post-Anubis), which proxies every request whole to the Authentik server with Host preserved — there is no forward_auth and no per-vhost block:

http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}

The embedded outpost inside the server process decides by Host: no session → the provider's authentication_flow is served in place; session valid → the request is forwarded to the provider's internal_host (http://caddy:8017), where per-vhost handle blocks reach the apps.

Lockstep rule: the Anubis guard set, the Caddy TLS edge, the :8017 routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients.

Live provider inventory (verified 2026-09-02), all attached to outpost 7e39655b-… ("authentik Embedded Outpost"), all with internal_host = http://caddy:8017, all three flows set, and each linked to an Application (see the provisioning rules below):

Provider pk	name	external_host	app slug
1	embedded-outpost	https://www.franksdank.cloud	caddy
2	search-proxy	https://search.franksdank.cloud	search
4	prometheus-proxy	https://prometheus.franksdank.cloud	prometheus
5	romm-proxy	https://romm.franksdank.cloud	romm
6	files-proxy	https://files.franksdank.cloud	files
7	loki-proxy	https://loki.franksdank.cloud/	loki
8	grafana-proxy	https://grafana.franksdank.cloud/	grafana-guard
10	seerr-proxy	https://seerr.franksdank.cloud/	seerr-guard

Provider 3 (hermes-proxy) still exists but is dormant — its vhost was decommissioned from Caddy on 2026-08-31. Provider 9 (grafana) and 12 (Seerr OIDC) are OAuth2 relying-party providers, not proxy providers; the outpost does not deserialize them. Trailing slashes on external_host (the runbook-recipe style) and bare origins both intercept correctly — the outpost matches on hostname.

Two provisioning rules, each verified live by an outage:

1. All three flows are required. A ProxyProvider created via raw API/ORM with any of authentication_flow/authorization_flow/invalidation_flow null wedges the embedded outpost's config fetch: the Rust side rejects the whole config JSON (serde: invalid type: null, expected a formatted UUID string), logs Failed to fetch outpost from API, retrying in 3 seconds forever, and everything behind the outpost dies at the next outpost reload — including hosts whose providers were fine (verified 2026-09-02 when three providers provisioned without any flows took down every guarded vhost after a server restart). Backfill all three flows (copy from pk 1: default-authentication-flow, default-provider-authorization-implicit-consent (the implicit-consent authorization flow, default-invalidation-flow) and the retry loop recovers on its next 3-second cycle. The silent killer: while the outpost process keeps its last good in-memory config, everything works — the outage only surfaces on restart/reload.
2. Every proxy provider needs an attached Application. The outpost resolves the post-login redirect through the provider's application; a provider with no app completes the login (authentik session established) but bounces to authentik's own user UI (/if/user/#/library) instead of the app. No error is logged anywhere — the SSO "works" and the app never appears. Create the Application with provider_id set (the UI does this automatically; raw API/ORM drives must do it explicitly).

OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (grafana, pk 9, confidential client) was created and verified end-to-end via ak shell ORM drives. This fork's OAuth2 provider implementation diverges from upstream in ways that bit us; all verified live:

• Application row required. OAuth2Provider.objects.create() does not create the single-table-inheritance Application row; without it the flow executor can never complete an authorize request. Create both rows.
• grant_types defaults to empty. Until the provider's grant_types contains "authorization_code" (enum value with underscore), the token endpoint rejects every exchange with invalid_request.
• Token endpoint is form-encoded + Basic-auth only. A JSON body makes parse_token_request see no grant_type → unsupported_grant_type; POST-body client secrets are not accepted — authenticate the confidential client with Authorization: Basic base64(client_id:client_secret).
• Provider resolution is by client_id query param on the global /application/o/authorize/ route; per-slug authorize paths do not exist in this build. Token/userinfo live at /application/o/token/ and /application/o/userinfo/?api=1.
• Property-mapping evaluator is broken → userinfo returns sub only. PropertyMappingEvaluator.evaluate(*args, **kwargs) forwards kwargs to BaseEvaluator.evaluate(expression_source), so the first context kwarg becomes the expression source instead of the context; every call raises a swallowed PropertyMappingExpressionException and the claim is skipped. Scope-grant filtering has the same class of bug: ScopeMapping has no provider FK (linkage lives in the ProviderPropertyMapping through model), so filter(provider=...) always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with {sub} only — email/groups/entitlements claims cannot be delivered by this build until the evaluator is patched or upgraded. Clients consuming this provider must key identity off sub.
• Scope mappings for this provider exist (grafana-email-scope, grafana-profile-scope, Python-syntax expressions) and will start emitting claims the moment the evaluator bug is fixed — no config change needed then.

LDAP outpost (Jellyfin auth, live 2026-09-01)

A fourth unit, dankclaw/authentik/quadlet/authentik-outpost-ldap.container → authentik-outpost-ldap.service, serves LDAP for Jellyfin's LDAP-Auth plugin. The image is pinned by its amd64 single-platform digest (ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4): podman 5.7.0 refuses the ghcr.io/goauthentik/ldap:2026.8.0 tag because the tag resolves to a nested manifest list (index → per-platform manifest lists) — unexpectedly received a manifest list instead of a manifest for a single image. Resolve once with podman manifest inspect <image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest' and pin that digest.

• Ports 389 (ldap) and 636 (ldaps) publish on the LAN; Jellyfin on Dankserver binds 192.168.0.143:389. The outpost token travels in AUTHENTIK_LDAP_OUTPOST_TOKEN from the host .env (git-ignored).

Authentik-side objects (provisioned via ak shell ORM)

Object	Value
LDAP provider	Jellyfin LDAP (pk 11), base DN dc=franksdank,dc=cloud
Bind DN	cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud
Service account	svc-jellyfin-ldap — passwordless/unusable; authenticates by token
Bind token	intent verification, non-expiring — its key is the LDAP bind password
RBAC role	svc-jellyfin-ldap-search carrying object-level permission authentik_providers_ldap.search_full_directory on the provider
Groups	jellyfin_users, jellyfin_admins
Application	jellyfin (slug jellyfin)

Hard-won rules, each verified live:

• Token intent is load-bearing. A bind token with intent=app_password is rejected by the flow's password stage with Invalid password; intent=verification passes. The outpost's direct binder executes the provider's authorization flow, and that password stage only accepts verification-intent tokens as static passwords.
• Search needs search_full_directory. Without the RBAC grant the outpost returns only the bind user itself — a (memberOf=…) filter then matches zero entries and the plugin logs Found no users matching X. The permission is object-level on the provider; grant it via an RBAC role and retest.
• ak shell import path for the provider model in 2026.8: from authentik.providers.ldap.models import LDAPProvider.

Configuration (secrets)

Two secrets live in the host .env (git-ignored; placeholders in .env.example):

• AUTHENTIK_PG_PASS — the PostgreSQL password. Generate: openssl rand -base64 32.
• AUTHENTIK_SECRET_KEY — the server's signing key. Generate: openssl rand -base64 48.

First-time setup

1. ./scripts/deploy.sh dankclaw — pulls, starts the 3 services (postgres health-gated), reloads Caddy.
2. Open https://auth.franksdank.cloud — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: Directory → Applications → add an application + a Proxy provider whose external_host is exactly the browser-facing origin (https://<host>), internal_host = http://caddy:8017, and all three flows set (see the provisioning pitfall above). Attach the provider to the embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an OAuth2/OpenID Connect provider, create the app with the correct redirect URI, and paste the client_id / client_secret into the app's SSO settings.

Operations

• Health: https://auth.franksdank.cloud/-/health/ (the server healthcheck probes this).
• Logs: podman logs authentik-server / podman logs authentik-worker.
• Backups: pg_dump the authentik-postgres database (the single source of truth). The /home/dank/Containers/authentik/data dir holds media + static files.
• Upgrades: bump the image tag in the three units under dankclaw/authentik/quadlet/, run ./scripts/deploy.sh dankclaw. Authentik migrates the schema on startup.

Self-serve user-settings flow (password change)

Users change their own account details through the default-user-settings-flow (https://auth.franksdank.cloud/if/flow/default-user-settings-flow/) — the target of the homepage footer's "account" link. It is a stage_configuration designated flow, so it renders every bound ConfigurableStage for the logged-in user.

Stages, in execution order (the FlowStageBinding.order field, not the M2M insertion order):

order	stage	type	what it does
20	default-user-settings	PromptStage	name / email / locale + password
100	default-user-settings-write	UserWriteStage	writes the prompt data to the user

Password change is a prompt field, not a stage. The UserWriteStage writes whatever the preceding PromptStage collected: for the key password it calls user.set_password(value) (stages/user_write/stage.py). So the fix is a password field on the default-user-settings prompt (field_key=password, type=password, required=False, order 204) — NOT a PasswordStage. A PasswordStage is an authentication stage: its validate_password runs authenticate() against the backends to check the current password; it never sets a new one. (A PasswordStage was added on 2026-09-25 and removed the same day for exactly this reason.)

Two ORM gotchas: Flow.stages.add() fails on this through model (Can't bulk create a multi-table inherited model — FlowStageBinding is MTI), so create the FlowStageBinding row directly with an explicit order; and a prompt field is a Prompt row linked through PromptStageField (prompt_stage + prompt; the order comes from Prompt.order).

A blank password must not set an empty password. set_password("") would make the account log in with no password, so a validation policy (default-user-settings-no-empty-password, bound to the prompt via PromptStageValidationPolicy) denies the form when prompt_data["password"] is "" and allows it when set. Verified with the PolicyEvaluator (empty → deny, set → allow, absent → allow) and end-to-end by driving UserWriteStageView.update_user on a throwaway user (new password valid, old invalid).

Outpost claim cache (proxysession) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to authentik_providers_proxy_proxysession whose session_data JSON holds the claims (groups included) that get served for that browser session until the row expires (observed TTLs 1–24 h per row). A session authorized before a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';

The silent re-auth path was proven end-to-end: after deleting qa-e2e-admin's rows, the e2e admin spec ran against the stale storage state and passed with no login prompt (2 tests, 2.9 s).

Guard-chain flip recipe (adding a guarded vhost)

Move a public vhost from reverse_proxy <app>:<port> onto the chain: client → TLS edge → Anubis :8923 → outpost decision → internal http://:8017 listener → Host-matched handler → real upstream.

Order (never reorder):

1. Provider first. Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app. Fleet convention for external_host is the exact browser-facing origin (the inventory table carries both bare origins and trailing slashes; routing matches on hostname, verified 2026-09-02). internal_host = http://caddy:8017, all three flows set, and an Application attached — the two provisioning rules above.
2. Route flip in ONE commit/deploy: the vhost becomes reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and inside the http://:8017 listener add @<svc> host <vhost> + handle @<svc> { reverse_proxy <upstream> }. Anubis guard set, Caddy TLS edge, :8017 routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).

Verification (all three):

• Unauthenticated probe from a client: curl -s https://<vhost>/ → HTTP 200 challenge page; grep the body for challenge|not a bot|anubis (~8 markers on a live challenge). A 502 here instead of the challenge = chain mis-wired.
• Post-auth routing proof: grep the LIVE config inside the container — podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile" — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0.
• Outpost reachability: podman exec authentik-server getent hosts caddy resolves the compose DNS alias (10.89.1.x) that internal_host dials.

Pitfalls: probing http://localhost:8017/ from the host returns HTTP 000 (connection refused) — the post-auth listener lives in the caddy container's netns; probe inside it or grep the config. Non-browser clients cannot solve the JS challenge — before flipping, audit service-to-service traffic: ingestion/push paths must target internal addresses (loki:3100), never a public hostname of a guarded service. Exempt-by-construction categories stay unguarded: the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth — document the exemption in the vhost comment.

Driving the ORM (ak shell)

All authentik object work happens through the server container's ORM shell — no repo change, no deploy (these providers are NOT repo-managed; capture new provider+application pairs as a blueprint or setup script to avoid drift).

ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF

• The heredoc IS the program: ak shell reads all of stdin and execs it (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven one).
• Bootstrap prints JSON log lines, then ### authentik shell (2026.8.3) and 229 objects imported automatically… — pipe through sed -n '/229 objects/,$p' to drop the noise.
• 229 models are pre-imported. Import paths (each wrong guess costs a failed exec): User, Group, Application, ApplicationEntitlement, PropertyMapping → authentik.core.models; ProxyProvider → authentik.providers.proxy.models (NOT authentik.outposts.models); LDAPProvider → authentik.providers.ldap.models; authentik.core.models.application does NOT exist as an import path; there is no authentik.policies.group module in this build.
• For live DB truth prefer the postgres container (podman exec authentik-postgres psql -U authentik -d authentik -At -c "select * from …") — query with select * and stop guessing column names: 2026.x renamed columns (Provider.objects.filter(type=…) raises FieldError — type moved to subclass tables; the outpost enabled column may not exist under that name; provider pk is oauth2provider_ptr_id).

User provisioning playbook (user-level account)

There is no User.create_user classmethod, and is_superuser is NOT a constructor kwarg (it is a property derived from group membership) — passing it raises TypeError: User() got unexpected keyword arguments:

from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...")        # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
    u.groups.add(Group.objects.get(name=g))

path defaults to users (regular). The access map (verified live 2026-10-04): there are NO per-app policy bindings on the proxy providers (all use the implicit-consent "Authorize Application" flow), so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed downstream —

• homepage-admins → Caddy RBAC for homepage /admin (X-Authentik-Groups regex gate) and the homepage admin panel; regular users get only category: user panels from services.json.
• openwebui_users / openwebui_admins → ApplicationEntitlements "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the "Open WebUI roles" property mapping → roles claim. Verify live with u.app_entitlements(app).
• jellyfin_users / jellyfin_admins → Jellyfin via the LDAP outpost (filter (memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)).
• Non-Admin Users = the plain-user marker group (precedents: alex, qa-e2e).
• EXCLUDE from a user-level account: homepage-admins, jellyfin_admins, openwebui_admins, authentik Admins, authentik Read-only.
• Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta) need no per-user group: any authenticated user passes the outpost.

Verify login behaviorally — authenticate() is the exact PasswordStage code path:

from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...")   # -> User
assert authenticate(request=None, username="ridley", password="nope") is None

Look flows up by slug, not name: Flow.objects.get(slug="default-authentication-flow") (its .name is "Welcome to authentik!" — .get(name=…) fails). Stage order: Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.

More field gotchas (each cost a failed exec): ApplicationEntitlement.application can be null (guard with getattr); LDAPProvider has search_mode/bind_mode/base_dn, lacks search_base/of_groups/ organization; OAuth2Provider has no .scopes; Outpost.config is an OutpostConfig object, not a dict; Policy has no uniform .policy_cls (use type(p).__name__); User has no .status.

Never rotate AUTHENTIK_SECRET_KEY to fix claim issues — it invalidates every outpost session fleet-wide; purge the stale proxysession rows instead (section above).

Probing the outpost from inside the netns

Rootless podman gives each container its own netns; host ss -ltn cannot see listeners inside other containers' namespaces — a missing listener on the host is an observation artifact, not a fault. Distroless images have no shell (podman exec authentik-server sh fails), so probe by joining the namespace:

podman run --rm --network container:<container> --entrypoint /bin/busybox \
  docker.io/library/busybox:latest \
  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8

• Quote URLs over ssh: the remote shell is zsh and bare ? in query strings triggers globbing (no matches found).
• Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding.
• podman inspect --format "{{.Config.NetMode}}" errors on this podman version — the field is .HostConfig.NetworkMode.

Failure modes

Symptom	Cause	Fix
Every vhost 502s at the outpost	authentik-server down	podman logs authentik-server; check postgres health
Login loops (302 → login → 302)	trusted_proxies / X-Forwarded-* mismatch	Confirm Caddy sets X-Forwarded-For; the outpost trusts private_ranges
"Invalid signature" on the outpost	AUTHENTIK_SECRET_KEY differs between server and worker	Both services read the same .env var; re-deploy
Postgres "password authentication failed"	AUTHENTIK_PG_PASS changed after first init	Reset via pg_ctl or re-init the data dir
"Request has been denied: Invalid next URL" at sign-in	Flow entered with an absolute cross-host next= (old redirect design)	Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie
/outpost.goauthentik.io/* 404s on every host	Any provider row with a null flow (authentication/authorization/invalidation) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload	Backfill all three flows on every provider; watch podman logs authentik-server for Failed to fetch outpost from API
Login completes but lands on authentik's user UI (/if/user/…) instead of the app	Proxy provider has no attached Application (outpost cannot resolve the post-login redirect)	Create the Application with provider_id set — no error is logged, so test with a fresh browser context
Off-LAN clients get "hostname could not be found"	Guarded host missing from the public DNS zone	Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer
Correct membership but one user still 403s at a header gate	Outpost claim cache: a proxysession row minted before the membership change serves old claims until expiry	Delete that user's proxysession rows (section above) — next visit re-authorizes silently

Cross-references

• Caddy — the reverse proxy carrying the :8016/:8017 auth-chain listeners
• Pi-hole — the LAN DNS that resolves auth.franksdank.cloud
• Jellyfin — consumes the LDAP outpost via its LDAP-Auth plugin
• Network Architecture — the flat naming scheme
• Research: .docaudit/research/auth.md — the full provider comparison