serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the	infrastructure authentik	2026-09-16

Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the homelab. It protects the Caddy vhosts that do not carry their own application-level authentication with its proxy outpost. Not wrapped: Gitea, OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) — they handle their own session. Grafana is the exception that is both: it is a guarded vhost (grafana-proxy, pk 8, locked to Authentik SSO) and an OAuth2 relying party (grafana, pk 9) (dankclaw/Caddyfile:122-132 dankclaw/Caddyfile:374-377; live provider table in Authentik Outpost Reference).

• Version: 2026.8.3 (pinned in dankclaw/authentik/quadlet/*.container the outpost's boot log reads "Booting authentik" … "version": "2026.8.3" 2026-10-18)
• Host: dankclaw (192.168.0.143)
• Name: auth.franksdank.cloud
• Stack: five quadlet units, no compose. authentik-postgres authentik-server, authentik-worker (deploy_authentik scripts/deploy-remote.sh:980-1004) plus two outposts — authentik-outpost (the Rust proxy outpost that actually serves /outpost.goauthentik.io/* deploy_authentik_outpost, scripts/deploy-remote.sh:1006-1018) and authentik-outpost-ldap (deploy_authentik_outpost_ldap scripts/deploy-remote.sh:1020-1032). All five report generated + active, Restart=always, WantedBy=default.target, with linger on (systemctl --user, 2026-10-18). The "3 compose services" wording is the pre-quadlet era, retired.
• LDAP outpost: serves LDAP for Jellyfin — see the section below.

Split off from this page: Authentik Processes owns the per-container facts for authentik-postgres, authentik-server and authentik-worker Authentik Outpost Reference owns the provider/flow/token lookup material — the live proxy-provider inventory, the proxy-provider provisioning rules, the OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow design, and the live database shape.

Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was recorded in .docaudit/research/auth.md, which no longer exists in the repo or in the host checkout (ls, 2026-10-18) (unverified 2026-10-18: the comparison itself). In short: full OIDC with single-logout, a web GUI for all configuration, an outpost that terminates auth for reverse-proxied apps, and a footprint that fits on a 3-user homelab — measured idle today podman stats --no-stream: postgres 233.8 MB + server 739.1 MB + worker 292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ 1.6 GB across five containers (the older "~1–1.5 GB" figure was the three-service stack). Redis was removed in 2025.10; the integration mechanism changed again in 2026.x (see Architecture).

Architecture

graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]

Who answers was re-measured 2026-10-18 from inside the caddy container with a tenant-matching Host: search.franksdank.cloud: http://authentik-server:9000/outpost.goauthentik.io/start → 302 into the authentication flow, while http://authentik-outpost:9000/… answers 404 for /outpost.goauthentik.io/* and for /if/flow/default-authentication-flow/. So the server process still terminates the proxy-provider decision, exactly as Caddy's :8016 → authentik-server:9000 hop assumes (dankclaw/Caddyfile:93-98). The authentik-outpost unit is installed, enabled (generated) and active, and it is the DB's authentik Embedded Outpost (7e39655b-…, nine providers attached) holding the websocket channel to core — but it serves no HTTP surface on the paths the chain needs. The unit header's stronger claim that the URL space "is served by the Rust worker binary, not the server" (dankclaw/authentik/quadlet/authentik-outpost.container:6-14) is not what this probe shows; do not point a new listener at authentik-outpost:9000 without re-measuring. Note also that a probe without a tenant Host (e.g. curl http://127.0.0.1:9000/…) 404s on paths that work — that is the Host-routing rule below, not an outage.

Two mechanisms were disproved live (2026-08-29/30) and one is the actual contract:

• The old /outpost.goauthentik.io/auth/caddy forward-auth endpoint does not exist in the 2026.8 URL tree — every unauthenticated subrequest got authentik's own 404 page.
• X-authentik-meta-outpost/-provider/-app headers are IGNORED for routing in this build: a request carrying Caddy's full meta-header set still 404s, while the same request with only a matching Host answers. The embedded outpost routes solely on the original Host, matched against each attached proxy provider's external_host.
• Consequence: every guarded host needs its own proxy provider (exact external_host), and :8016 is a single uniform reverse_proxy to authentik-server:9000 that preserves Host. Unauthenticated → login flow served in place; authenticated → forwarded to the provider's internal_host (http://caddy:8017), which routes straight to the app. The chain terminates there — nothing dials back into Authentik from :8017.

The stack's processes

Per-container facts — unit path, image and role, port, run-as, data directory, health probe — are owned by Authentik Processes.

Caddy integration

Guarded vhosts reach Caddy's internal listener :8016 (post-Anubis), which proxies every request whole to the Authentik server with Host preserved — there is no forward_auth and no per-vhost block:

http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}

The outpost decides by Host: no session → the provider's authentication_flow is served in place; session valid → the request is forwarded to the provider's internal_host (http://caddy:8017), where per-vhost handle blocks reach the apps.

Lockstep rule: the Anubis guard set, the Caddy TLS edge, the :8017 routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients.

The live proxy-provider inventory (the per-vhost external_host → app mapping) the two provisioning rules the outpost enforces, and this fork's OAuth2 relying-party quirks are reference lookup: Authentik Outpost Reference.

LDAP outpost (Jellyfin auth, live 2026-09-01)

The second outpost — dankclaw/authentik/quadlet/authentik-outpost-ldap.container → authentik-outpost-ldap.service, one of the five units in this stack — serves LDAP for Jellyfin's LDAP-Auth plugin (enabled=generated, active Restart=always, container up, 2026-10-18). The image is pinned by its amd64 single-platform digest ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9 (dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26, matching the running image in podman ps, 2026-10-18). The digest in this page before 2026-10-18 (…fa33bc8f…) is superseded. The reason for pinning a digest at all: podman 5.7.0 refuses the plain tag because it resolves to a nested manifest list (index → per-platform manifest lists) — unexpectedly received a manifest list instead of a manifest for a single image — first hit on :2026.8.0, still the rule at the current 2026.8.3 pin. Resolve once with podman manifest inspect <image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest' and pin that digest.

• Ports 389 (ldap) and 636 (ldaps) publish on the LAN (PublishPort=389:3389, 636:6636 dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31; ss -ltnp shows 0.0.0.0:389 and 0.0.0.0:636, 2026-10-18) and Dankserver can reach them — a bare TCP connect from Dankserver to 192.168.0.143:389 succeeds (2026-10-18). The outpost token travels in AUTHENTIK_LDAP_OUTPOST_TOKEN from the host dankclaw/.env (git-ignored).

The objects behind it — LDAP provider, bind DN, service account and bind token, RBAC grant, groups, application slug — and the hard-won bind rules are in Authentik Outpost Reference.

Configuration (secrets)

Four variables live in the host env file the units read, /home/dank/serverAppConfigs/dankclaw/.env (git-ignored; the quadlets expand ${…} from it via EnvironmentFile):

• AUTHENTIK_PG_PASS — the PostgreSQL password. Generate: openssl rand -base64 32.
• AUTHENTIK_SECRET_KEY — the server's signing key. Generate: openssl rand -base64 48.
• AUTHENTIK_OUTPOST_TOKEN — proxy outpost token (dankclaw/authentik/quadlet/authentik-outpost.container:34).
• AUTHENTIK_LDAP_OUTPOST_TOKEN — LDAP outpost token (…/authentik-outpost-ldap.container:29).

Only the first two have placeholders in dankclaw/.env.example; the two outpost tokens have no example entry (there is no repository-root .env.example — find, 2026-10-18). Rotating AUTHENTIK_SECRET_KEY fleet-wide invalidates every outpost session (see the ORM playbook).

First-time setup

1. ./scripts/deploy.sh dankclaw --service authentik-server (any of authentik-postgres|authentik-server|authentik-worker runs deploy_authentik, scripts/deploy-remote.sh:980-1004) — installs the three core quadlets, starts postgres health-gated (wait_container_healthy authentik-postgres 120), then the server (300 s health wait) and the worker. The outposts are separate handlers: --service authentik-outpost and --service authentik-outpost-ldap (scripts/deploy-remote.sh:1006-1032 dispatch at :1579-1588).
2. Open https://auth.franksdank.cloud — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: Directory → Applications → add an application + a Proxy provider whose external_host is exactly the browser-facing origin (https://<host>), internal_host = http://caddy:8017, and all three flows set (see the provisioning rules in Authentik Outpost Reference). Attach the provider to the embedded outpost.
4. For apps with native OIDC: add an OAuth2/OpenID Connect provider, create the app with the correct redirect URI, and paste the client_id / client_secret into the app's SSO settings. The relying parties that exist today are grafana (pk 9), Seerr OIDC (pk 12) and Open WebUI (pk 14) Gitea has no provider row at all (live provider table, 2026-10-18) — it authenticates itself and stays off the chain by design (dankclaw/Caddyfile:110-120).

Operations

• Health: there is no HTTP health path on this build — /-/health/, /-/live/ and /api/-/health/ all answer 404, on 127.0.0.1:9000 and through https://auth.franksdank.cloud alike (curl, 2026-10-18). The container's own probe is Django-side: HealthCmd=python /manage.py check --database default (dankclaw/authentik/quadlet/authentik-server.container:36), and podman inspect authentik-server reports healthy. A usable liveness URL is a flow page: https://auth.franksdank.cloud/if/flow/default-user-settings-flow/ → 200 (2026-10-18). The earlier "the server healthcheck probes /-/health/" line was wrong in both halves.
• Logs: podman logs authentik-server / podman logs authentik-worker.
• Backups: pg_dump the authentik-postgres database (the single source of truth). The /home/dank/Containers/authentik/data dir holds media + static files (mounted by all three app units).
• Upgrades: bump the tag in all three app units under dankclaw/authentik/quadlet/ and re-deploy; authentik-outpost carries the same tag (it runs the same image, Exec=worker) and authentik-outpost-ldap is digest-pinned, so its pin must be re-resolved for the new version (see the LDAP section) — bumping only the tag would leave the LDAP outpost behind. Authentik migrates the schema on startup.

Outpost claim cache (proxysession) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to authentik_providers_proxy_proxysession whose session_data JSON holds the claims (groups included) that get served for that browser session until the row expires (observed TTLs 1–24 h row). A session authorized before a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';

The silent re-auth path was proven end-to-end: after deleting qa-e2e-admin's rows, the e2e admin spec ran against the stale storage state and passed with no login prompt (2 tests, 2.9 s).

Guard-chain flip recipe (adding a guarded vhost)

Move a public vhost from reverse_proxy <app>:<port> onto the chain: client → TLS edge → Anubis :8923 → outpost decision → internal http://:8017 listener → Host-matched handler → real upstream.

Order (never reorder):

1. Provider first. Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app. Fleet convention for external_host is the exact browser-facing origin (the inventory table carries both bare origins and trailing slashes; routing matches on hostname, verified 2026-09-02). internal_host = http://caddy:8017, all three flows set, and an Application attached — the two provisioning rules in Authentik Outpost Reference.
2. Route flip in ONE commit/deploy: the vhost becomes reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and inside the http://:8017 listener add @<svc> host <vhost> + handle @<svc> { reverse_proxy <upstream> }. Anubis guard set, Caddy TLS edge, :8017 routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).

Verification (all three):

• Unauthenticated probe from a client: curl -s https://<vhost>/ → HTTP 200 challenge page; grep the body for challenge|not a bot|anubis (~8 markers on a live challenge). A 502 here instead of the challenge = chain mis-wired.
• Post-auth routing proof: grep the LIVE config inside the container — podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile" — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0.
• Outpost reachability: podman exec authentik-server getent hosts caddy → 10.89.1.8 caddy.dns.podman (2026-10-18) — that is Caddy's address on the dankclaw_default podman network, the alias internal_host dials. It is a netavark allocation, not a pinned address and not a compose artifact (the compose stack is gone); never hard-code it.

Pitfalls: probing http://localhost:8017/ from the host returns HTTP 000 (connection refused) — the post-auth listener lives in the caddy container's netns; probe inside it or grep the config. Non-browser clients cannot solve the JS challenge — before flipping, audit service-to-service traffic: ingestion/push paths must target internal addresses (loki:3100) never a public hostname of a guarded service. Exempt-by-construction categories stay unguarded: the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth — document the exemption in the vhost comment.

Driving the ORM (ak shell)

All authentik object work happens through the server container's ORM shell — no repo change, no deploy (these providers are NOT repo-managed; capture new provider+application pairs as a blueprint or setup script to avoid drift).

ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF

• The heredoc IS the program: ak shell reads all of stdin and execs it (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven one).
• Bootstrap prints JSON log lines, then a shell banner and an "NNN objects imported automatically…" line — pipe through sed -n '/objects imported/,$p' to drop the noise (unverified 2026-10-18: the exact banner text and the 229-object count — no ak shell was driven against the live auth service for this audit).
• Import paths (each wrong guess costs a failed exec): User, Group Application, ApplicationEntitlement, PropertyMapping → authentik.core.models; ProxyProvider → authentik.providers.proxy.models (NOT authentik.outposts.models) LDAPProvider → authentik.providers.ldap.models authentik.core.models.application does NOT exist as an import path; there is no authentik.policies.group module in this build.
  • (unverified 2026-10-18: these module paths were not re-executed — importing outside a configured Django shell fails with ImproperlyConfigured)*
• The live database's table and column names (the 2026.x renames, the MTI parent keys, the read-only psql audit recipe) are reference lookup: Authentik Outpost Reference.

User provisioning playbook (user-level account)

There is no User.create_user classmethod, and is_superuser is NOT a constructor kwarg (it is a property derived from group membership) — passing it raises TypeError: User() got unexpected keyword arguments:

from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...")        # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
    u.groups.add(Group.objects.get(name=g))

path defaults to users (regular). The access map (re-checked against the live DB 2026-10-18): every one of the nine proxy providers carries the same authorization flow UUID, default-provider-authorization-implicit-consent (name "Authorize Application"), i.e. there are no per-app policy bindings to delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed downstream —

• homepage-admins → Caddy RBAC for homepage /admin (X-Authentik-Groups regex gate) and the homepage admin panel; regular users get only category: user panels from services.json.
• openwebui_users / openwebui_admins → ApplicationEntitlements "Open WebUI Users" and "Open WebUI Admins", both rows attached to the Open WebUI application (authentik_core_applicationentitlement 2026-10-18), consumed by the "Open WebUI roles" property mapping → roles claim (unverified 2026-10-18: the group→entitlement binding and the mapping expression were not read; the mapping itself is not visible under that name in authentik_core_propertymapping). Verify live with u.app_entitlements(app).
• jellyfin_users / jellyfin_admins → Jellyfin via the LDAP outpost (filter (memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)).
• Non-Admin Users = the plain-user marker group (precedents: qa-e2e, and every human account).
• The human-user template (ridley, kassie, alex, artem, kirby, ry, nic, debo, freya, mitch, eric). The pks in this page are confirmed against authentik_core_user on 2026-10-18: ridley 13, kassie 14, alex 15, artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 — and every one carries the <user>@franksdank.cloud address the template prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
  • (unverified 2026-10-18)*. Template: username + <user>@franksdank.cloud + display name + is_active=True + the password set via set_password, membership in Non-Admin Users + jellyfin_users + openwebui_users. alex was recreated from scratch on 2026-10-08 into this shape (the old row had an empty email and no openwebui_users; audit first — the row carried no MFA device, token, entitlement, or proxy session, and events survive a user delete because Event.user is a denormalized JSON field, not a FK).
• EXCLUDE from a user-level account: homepage-admins, jellyfin_admins openwebui_admins, authentik Admins, authentik Read-only.
• Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta) need no per-user group: any authenticated user passes the outpost.

Verify login behaviorally — authenticate() is the exact PasswordStage code path:

from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...")   # -> User
assert authenticate(request=None, username="ridley", password="nope") is None

Look flows up by slug, not name: Flow.objects.get(slug="default-authentication-flow") (its .name is "Welcome to authentik!" — .get(name=…) fails; both confirmed from authentik_flows_flow, 2026-10-18). Stage order, read from authentik_flows_flowstagebinding the same day: default-authentication-identification(10) → default-authentication-password(20) → default-authentication-mfa-validation(30; no-op with no enrolled device) → default-authentication-login(100). With no MFA enrolled, nothing else blocks.

More field gotchas (each cost a failed exec): ApplicationEntitlement.application can be null (guard with getattr); LDAPProvider has search_mode/bind_mode/base_dn and lacks search_base/of_groups/ organization — the backing table (authentik_providers_ldap_ldapprovider) has exactly provider_ptr_id base_dn, certificate_id, tls_server_name, gid_start_number uid_start_number, search_mode, bind_mode, mfa_support (information_schema.columns, 2026-10-18); OAuth2Provider has no .scopes (scopes are linked through ProviderPropertyMapping, which is why filter(provider=…) on a scope mapping is empty); Outpost.config is an OutpostConfig object, not a dict; Policy has no uniform .policy_cls (use type(p).__name__); User has no .status.

Never rotate AUTHENTIK_SECRET_KEY to fix claim issues — it invalidates every outpost session fleet-wide; purge the stale proxysession rows instead (section above).

Probing the outpost from inside the netns

Rootless podman gives each container its own netns; host ss -ltn cannot see listeners inside other containers' namespaces — a missing listener on the host is an observation artifact, not a fault. Distroless images have no shell (podman exec authentik-server sh fails), so probe by joining the namespace:

podman run --rm --network container:<container> --entrypoint /bin/busybox \
  docker.io/library/busybox:latest \
  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8

• Quote URLs over ssh: the remote shell is zsh and bare ? in query strings triggers globbing (no matches found).
• Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding.
• podman inspect --format "{{.Config.NetMode}}" errors on this podman version — the field is .HostConfig.NetworkMode.

Failure modes

Symptom	Cause	Fix
Every vhost 502s at the outpost	authentik-server down	podman logs authentik-server; check postgres health
Login loops (302 → login → 302)	trusted_proxies / X-Forwarded-* mismatch	Confirm Caddy sets X-Forwarded-For; the outpost trusts private_ranges
"Invalid signature" on the outpost	AUTHENTIK_SECRET_KEY differs between server and worker	Both services read the same .env var; re-deploy
Postgres "password authentication failed"	AUTHENTIK_PG_PASS changed after first init	Reset via pg_ctl or re-init the data dir
"Request has been denied: Invalid next URL" at sign-in	Flow entered with an absolute cross-host next= (old redirect design)	Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie
/outpost.goauthentik.io/* 404s on every host	Any provider row with a null flow (authentication/authorization/invalidation) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload	Backfill all three flows on every provider; watch podman logs authentik-server for Failed to fetch outpost from API
Login completes but lands on authentik's user UI (/if/user/…) instead of the app	Proxy provider has no attached Application (outpost cannot resolve the post-login redirect)	Create the Application with provider_id set — no error is logged, so test with a fresh browser context
Off-LAN clients get "hostname could not be found"	Guarded host missing from the public DNS zone	Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer
Correct membership but one user still 403s at a header gate	Outpost claim cache: a proxysession row minted before the membership change serves old claims until expiry	Delete that user's proxysession rows (section above) — next visit re-authorizes silently

Cross-references

• Caddy — the reverse proxy carrying the :8016/:8017 auth-chain listeners
• Pi-hole — the LAN DNS that resolves auth.franksdank.cloud
• Jellyfin — consumes the LDAP outpost via its LDAP-Auth plugin
• Network Architecture — the flat naming scheme
• Research: .docaudit/research/auth.md — the full provider comparison
  • (the file is gone from the repo and from the host checkout as of 2026-10-18)*