serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the	infrastructure authentik	2026-09-16

Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the homelab. It protects the Caddy vhosts that do not carry their own application-level authentication with its proxy outpost. Not wrapped: Gitea, OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) — they handle their own session. Grafana is the exception that is both: it is a guarded vhost (grafana-proxy, pk 8, locked to Authentik SSO) and an OAuth2 relying party (grafana, pk 9) (dankclaw/Caddyfile:122-132, dankclaw/Caddyfile:374-377; live provider table below).

• Version: 2026.8.3 (pinned in dankclaw/authentik/quadlet/*.container; the outpost's boot log reads "Booting authentik" … "version": "2026.8.3", 2026-10-18)
• Host: dankclaw (192.168.0.143)
• Name: auth.franksdank.cloud
• Stack: five quadlet units, no compose. authentik-postgres, authentik-server, authentik-worker (deploy_authentik, scripts/deploy-remote.sh:980-1004) plus two outposts — authentik-outpost (the Rust proxy outpost that actually serves /outpost.goauthentik.io/*, deploy_authentik_outpost, scripts/deploy-remote.sh:1006-1018) and authentik-outpost-ldap (deploy_authentik_outpost_ldap, scripts/deploy-remote.sh:1020-1032). All five report generated + active, Restart=always, WantedBy=default.target, with linger on (systemctl --user, 2026-10-18). The "3 compose services" wording is the pre-quadlet era, retired.
• LDAP outpost: serves LDAP for Jellyfin — see the section below.

Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was recorded in .docaudit/research/auth.md, which no longer exists in the repo or in the host checkout (ls, 2026-10-18) (unverified 2026-10-18: the comparison itself). In short: full OIDC with single-logout, a web GUI for all configuration, an outpost that terminates auth for reverse-proxied apps, and a footprint that fits on a 3-user homelab — measured idle today podman stats --no-stream: postgres 233.8 MB + server 739.1 MB + worker 292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ 1.6 GB across five containers (the older "~1–1.5 GB" figure was the three-service stack). Redis was removed in 2025.10; the integration mechanism changed again in 2026.x (see Architecture).

Architecture

graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]

Who answers was re-measured 2026-10-18 from inside the caddy container with a tenant-matching Host: search.franksdank.cloud: http://authentik-server:9000/outpost.goauthentik.io/start → 302 into the authentication flow, while http://authentik-outpost:9000/… answers 404 for /outpost.goauthentik.io/* and for /if/flow/default-authentication-flow/. So the server process still terminates the proxy-provider decision, exactly as Caddy's :8016 → authentik-server:9000 hop assumes (dankclaw/Caddyfile:93-98). The authentik-outpost unit is installed, enabled (generated) and active, and it is the DB's authentik Embedded Outpost (7e39655b-…, nine providers attached) holding the websocket channel to core — but it serves no HTTP surface on the paths the chain needs. The unit header's stronger claim that the URL space "is served by the Rust worker binary, not the server" (dankclaw/authentik/quadlet/authentik-outpost.container:6-14) is not what this probe shows; do not point a new listener at authentik-outpost:9000 without re-measuring. Note also that a probe without a tenant Host (e.g. curl http://127.0.0.1:9000/…) 404s on paths that work — that is the Host-routing rule below, not an outage.

Two mechanisms were disproved live (2026-08-29/30) and one is the actual contract:

• The old /outpost.goauthentik.io/auth/caddy forward-auth endpoint does not exist in the 2026.8 URL tree — every unauthenticated subrequest got authentik's own 404 page.
• X-authentik-meta-outpost/-provider/-app headers are IGNORED for routing in this build: a request carrying Caddy's full meta-header set still 404s, while the same request with only a matching Host answers. The embedded outpost routes solely on the original Host, matched against each attached proxy provider's external_host.
• Consequence: every guarded host needs its own proxy provider (exact external_host), and :8016 is a single uniform reverse_proxy to authentik-server:9000 that preserves Host. Unauthenticated → login flow served in place; authenticated → forwarded to the provider's internal_host (http://caddy:8017), which routes straight to the app. The chain terminates there — nothing dials back into Authentik from :8017.

The three compose services

Service	Role	Image
authentik-postgres	PostgreSQL 16 (users, flows, sessions, events)	docker.io/library/postgres:16-alpine
authentik-server	Core + admin API + the embedded Caddy outpost (port 9000)	ghcr.io/goauthentik/server:2026.8.3
authentik-worker	Background tasks (event processing, provider sync, flows)	ghcr.io/goauthentik/server:2026.8.3

• Postgres is the only stateful store. Sessions, caching, and WebSockets moved to PostgreSQL in 2025.10 (Redis removed).
• Server and worker run the same image; the command selects the role (server / worker).
• The worker runs as 0:0 (rootless: container root maps to the host user that owns the data dir).
• Data persists under /home/dank/Containers/authentik/{data,postgres} (bind mounts, repo policy — no named volumes; moved from /home/dank/authentik/ in the 2026-09-01 quadlet standardization).

Caddy integration

Guarded vhosts reach Caddy's internal listener :8016 (post-Anubis), which proxies every request whole to the Authentik server with Host preserved — there is no forward_auth and no per-vhost block:

http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}

The outpost decides by Host: no session → the provider's authentication_flow is served in place; session valid → the request is forwarded to the provider's internal_host (http://caddy:8017), where per-vhost handle blocks reach the apps.

Lockstep rule: the Anubis guard set, the Caddy TLS edge, the :8017 routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients.

Live provider inventory (read from the authentik-postgres container with read-only SELECTs, 2026-10-18). Nine proxy providers, all attached to outpost 7e39655b-58d3-4fe6-a80a-513f31051217 ("authentik Embedded Outpost", 9 members in authentik_outposts_outpost_providers), all with internal_host = http://caddy:8017, all carrying the same three flow UUIDs (default-authentication-flow, default-provider-authorization-implicit-consent, default-provider-invalidation-flow), and each backed by an Application (the slug column below):

Provider pk	name	external_host	app slug
1	embedded-outpost	https://www.franksdank.cloud	caddy
2	search-proxy	https://search.franksdank.cloud	search
4	prometheus-proxy	https://prometheus.franksdank.cloud	prometheus
5	romm-proxy	https://romm.franksdank.cloud	romm
6	files-proxy	https://files.franksdank.cloud	files
7	loki-proxy	https://loki.franksdank.cloud/	loki
8	grafana-proxy	https://grafana.franksdank.cloud/	grafana-guard
10	seerr-proxy	https://seerr.franksdank.cloud/	seerr-guard
13	marreta-proxy	https://marreta.franksdank.cloud	marreta

That is exactly the Caddyfile's guarded set (dankclaw/Caddyfile:283-423), so the lockstep rule holds today. Two corrections against the 2026-09-02 table: pk 13 (marreta-proxy) was added when Marreta went onto the chain, and pk 3 (hermes-proxy) no longer exists at all — select … where id=3 returns zero rows, so it is deleted, not "dormant"; its vhost survived but is exempt-by-construction (dankclaw/Caddyfile:179-191). OAuth2 relying parties, not proxy providers, so the proxy outpost does not deserialize them: pk 9 grafana (grant_types = {authorization_code}), pk 12 Seerr OIDC (grant_types = {} — empty, i.e. still in the state the token-endpoint rule below rejects), and pk 14 Open WebUI (grant_types = {authorization_code,refresh_token}). Trailing slashes on external_host (the runbook-recipe style) and bare origins both intercept correctly — the outpost matches on hostname.

Two provisioning rules, each verified live by an outage:

1. All three flows are required. A ProxyProvider created via raw API/ORM with any of authentication_flow/authorization_flow/invalidation_flow null wedges the embedded outpost's config fetch: the Rust side rejects the whole config JSON (serde: invalid type: null, expected a formatted UUID string), logs Failed to fetch outpost from API, retrying in 3 seconds forever, and everything behind the outpost dies at the next outpost reload — including hosts whose providers were fine (verified 2026-09-02 when three providers provisioned without any flows took down every guarded vhost after a server restart). Backfill all three flows (copy from pk 1: default-authentication-flow, default-provider-authorization-implicit-consent (the implicit-consent authorization flow, default-invalidation-flow) and the retry loop recovers on its next 3-second cycle. The silent killer: while the outpost process keeps its last good in-memory config, everything works — the outage only surfaces on restart/reload.
2. Every proxy provider needs an attached Application. The outpost resolves the post-login redirect through the provider's application; a provider with no app completes the login (authentik session established) but bounces to authentik's own user UI (/if/user/#/library) instead of the app. No error is logged anywhere — the SSO "works" and the app never appears. Create the Application with provider_id set (the UI does this automatically; raw API/ORM drives must do it explicitly).

OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (grafana, pk 9, confidential client) was created and verified end-to-end via ak shell ORM drives. This fork's OAuth2 provider implementation diverges from upstream in ways that bit us; all verified live:

• Application row required. OAuth2Provider.objects.create() does not create the single-table-inheritance Application row; without it the flow executor can never complete an authorize request. Create both rows.
• grant_types defaults to empty. Until the provider's grant_types contains "authorization_code" (enum value with underscore), the token endpoint rejects every exchange with invalid_request.
• Token endpoint is form-encoded + Basic-auth only. A JSON body makes parse_token_request see no grant_type → unsupported_grant_type; POST-body client secrets are not accepted — authenticate the confidential client with Authorization: Basic base64(client_id:client_secret).
• Provider resolution is by client_id query param on the global /application/o/authorize/ route; per-slug authorize paths do not exist in this build. Token/userinfo live at /application/o/token/ and /application/o/userinfo/?api=1.
• Property-mapping evaluator is broken → userinfo returns sub only. PropertyMappingEvaluator.evaluate(*args, **kwargs) forwards kwargs to BaseEvaluator.evaluate(expression_source), so the first context kwarg becomes the expression source instead of the context; every call raises a swallowed PropertyMappingExpressionException and the claim is skipped. Scope-grant filtering has the same class of bug: ScopeMapping has no provider FK (linkage lives in the ProviderPropertyMapping through model), so filter(provider=...) always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with {sub} only — email/groups/entitlements claims cannot be delivered by this build until the evaluator is patched or upgraded. Clients consuming this provider must key identity off sub.
• Scope mappings for this provider exist — grafana-email-scope and grafana-profile-scope are both rows in authentik_core_propertymapping (read-only SELECT, 2026-10-18) — and will start emitting claims the moment the evaluator bug is fixed; no config change is needed then.

LDAP outpost (Jellyfin auth, live 2026-09-01)

The second outpost — dankclaw/authentik/quadlet/authentik-outpost-ldap.container → authentik-outpost-ldap.service, one of the five units in this stack — serves LDAP for Jellyfin's LDAP-Auth plugin (enabled=generated, active, Restart=always, container up, 2026-10-18). The image is pinned by its amd64 single-platform digest ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9 (dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26, matching the running image in podman ps, 2026-10-18). The digest in this page before 2026-10-18 (…fa33bc8f…) is superseded. The reason for pinning a digest at all: podman 5.7.0 refuses the plain tag because it resolves to a nested manifest list (index → per-platform manifest lists) — unexpectedly received a manifest list instead of a manifest for a single image — first hit on :2026.8.0, still the rule at the current 2026.8.3 pin. Resolve once with podman manifest inspect <image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest' and pin that digest.

• Ports 389 (ldap) and 636 (ldaps) publish on the LAN (PublishPort=389:3389, 636:6636, dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31; ss -ltnp shows 0.0.0.0:389 and 0.0.0.0:636, 2026-10-18) and Dankserver can reach them — a bare TCP connect from Dankserver to 192.168.0.143:389 succeeds (2026-10-18). The outpost token travels in AUTHENTIK_LDAP_OUTPOST_TOKEN from the host dankclaw/.env (git-ignored).

Authentik-side objects (provisioned via ak shell ORM)

Read from the live database (read-only SELECTs against the authentik-postgres container, 2026-10-18):

Object	Value
LDAP provider	Jellyfin LDAP (pk 11), base DN dc=franksdank,dc=cloud, search_mode=direct, bind_mode=direct
Bind DN	cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud (unverified 2026-10-18: base DN confirmed in the DB, the ou=users path was not probed — an LDAP bind needs the bind secret)
Service account	svc-jellyfin-ldap (user pk 8). It is not passwordless: its password field is a real hash, not Django's !-prefixed unusable marker (starts_with(password,'!') → f) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow (unverified 2026-10-18: whether anyone holds that password)
Bind token	exactly one row for pk 8, intent=app_password, non-expiring — the page previously recorded intent=verification; the live value is app_password
RBAC role	svc-jellyfin-ldap-search, carrying exactly one object-level permission authentik_providers_ldap.search_full_directory (verified via guardian_roleobjectpermission)
Groups	jellyfin_users, jellyfin_admins (both exist in authentik_core_group)
Application	name Jellyfin, slug jellyfin-ldap (the page previously said slug jellyfin, which does not exist)

Hard-won rules (the first is now contested by the live database):

• Token intent: contested, do not trust either value blindly. This page long asserted that a bind token with intent=app_password is rejected by the flow's password stage with Invalid password and only intent=verification passes. The live token for svc-jellyfin-ldap is app_password (authentik_core_token, 2026-10-18), so either the rule is wrong or the current Jellyfin bind path is not that token. Nothing observable settles it today: the LDAP outpost's retained log holds only its 5-minute initialized direct binder / Update providers config cycles and no bind activity (unverified 2026-10-18: no bind was performed — reproducing it requires the bind secret). Re-verify with a real bind before changing the token's intent either way.
• Search needs search_full_directory. Without the RBAC grant the outpost returns only the bind user itself — a (memberOf=…) filter then matches zero entries and the plugin logs Found no users matching X. The permission is object-level on the provider; grant it via an RBAC role and retest.
• ak shell import path for the provider model in 2026.8: from authentik.providers.ldap.models import LDAPProvider. (unverified 2026-10-18: importing it outside a configured Django shell fails with ImproperlyConfigured, and no ak shell was driven against the live auth service for this check)

Configuration (secrets)

Four variables live in the host env file the units read, /home/dank/serverAppConfigs/dankclaw/.env (git-ignored; the quadlets expand ${…} from it via EnvironmentFile):

• AUTHENTIK_PG_PASS — the PostgreSQL password. Generate: openssl rand -base64 32.
• AUTHENTIK_SECRET_KEY — the server's signing key. Generate: openssl rand -base64 48.
• AUTHENTIK_OUTPOST_TOKEN — proxy outpost token (dankclaw/authentik/quadlet/authentik-outpost.container:34).
• AUTHENTIK_LDAP_OUTPOST_TOKEN — LDAP outpost token (…/authentik-outpost-ldap.container:29).

Only the first two have placeholders in dankclaw/.env.example; the two outpost tokens have no example entry (there is no repository-root .env.example — find, 2026-10-18). Rotating AUTHENTIK_SECRET_KEY fleet-wide invalidates every outpost session (see the ORM playbook).

First-time setup

1. ./scripts/deploy.sh dankclaw --service authentik-server (any of authentik-postgres|authentik-server|authentik-worker runs deploy_authentik, scripts/deploy-remote.sh:980-1004) — installs the three core quadlets, starts postgres health-gated (wait_container_healthy authentik-postgres 120), then the server (300 s health wait) and the worker. The outposts are separate handlers: --service authentik-outpost and --service authentik-outpost-ldap (scripts/deploy-remote.sh:1006-1032, dispatch at :1579-1588).
2. Open https://auth.franksdank.cloud — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: Directory → Applications → add an application + a Proxy provider whose external_host is exactly the browser-facing origin (https://<host>), internal_host = http://caddy:8017, and all three flows set (see the provisioning pitfall above). Attach the provider to the embedded outpost.
4. For apps with native OIDC: add an OAuth2/OpenID Connect provider, create the app with the correct redirect URI, and paste the client_id / client_secret into the app's SSO settings. The relying parties that exist today are grafana (pk 9), Seerr OIDC (pk 12) and Open WebUI (pk 14); Gitea has no provider row at all (live provider table, 2026-10-18) — it authenticates itself and stays off the chain by design (dankclaw/Caddyfile:110-120).

Operations

• Health: there is no HTTP health path on this build — /-/health/, /-/live/ and /api/-/health/ all answer 404, on 127.0.0.1:9000 and through https://auth.franksdank.cloud alike (curl, 2026-10-18). The container's own probe is Django-side: HealthCmd=python /manage.py check --database default (dankclaw/authentik/quadlet/authentik-server.container:36), and podman inspect authentik-server reports healthy. A usable liveness URL is a flow page: https://auth.franksdank.cloud/if/flow/default-user-settings-flow/ → 200 (2026-10-18). The earlier "the server healthcheck probes /-/health/" line was wrong in both halves.
• Logs: podman logs authentik-server / podman logs authentik-worker.
• Backups: pg_dump the authentik-postgres database (the single source of truth). The /home/dank/Containers/authentik/data dir holds media + static files (mounted by all three app units).
• Upgrades: bump the tag in all three app units under dankclaw/authentik/quadlet/ and re-deploy; authentik-outpost carries the same tag (it runs the same image, Exec=worker) and authentik-outpost-ldap is digest-pinned, so its pin must be re-resolved for the new version (see the LDAP section) — bumping only the tag would leave the LDAP outpost behind. Authentik migrates the schema on startup.

Self-serve user-settings flow (password change)

Users change their own account details through the default-user-settings-flow (https://auth.franksdank.cloud/if/flow/default-user-settings-flow/) — the target of the homepage footer's "account" link. It is a stage_configuration designated flow, so it renders every bound ConfigurableStage for the logged-in user.

Stages, in execution order (the FlowStageBinding.order field, not the M2M insertion order):

order	stage	type	what it does
20	default-user-settings	PromptStage	name / email / locale + password
100	default-user-settings-write	UserWriteStage	writes the prompt data to the user

Password change is a prompt field, not a stage. The UserWriteStage writes whatever the preceding PromptStage collected: for the key password it calls user.set_password(value) (stages/user_write/stage.py). So the fix is a password field on the default-user-settings prompt (field_key=password, type=password, required=False, order 204) — NOT a PasswordStage. A PasswordStage is an authentication stage: its validate_password runs authenticate() against the backends to check the current password; it never sets a new one. (A PasswordStage was added on 2026-09-25 and removed the same day for exactly this reason.)

Two ORM gotchas: Flow.stages.add() fails on this through model (Can't bulk create a multi-table inherited model — FlowStageBinding is MTI), so create the FlowStageBinding row directly with an explicit order; and a prompt field is a Prompt row linked through PromptStageField (prompt_stage + prompt; the order comes from Prompt.order).

A blank password must not set an empty password. set_password("") would make the account log in with no password, so a validation policy (default-user-settings-no-empty-password, bound to the prompt via PromptStageValidationPolicy) denies the form when prompt_data["password"] is "" and allows it when set. Verified with the PolicyEvaluator (empty → deny, set → allow, absent → allow) and end-to-end by driving UserWriteStageView.update_user on a throwaway user (new password valid, old invalid).

Outpost claim cache (proxysession) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to authentik_providers_proxy_proxysession whose session_data JSON holds the claims (groups included) that get served for that browser session until the row expires (observed TTLs 1–24 h per row). A session authorized before a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';

The silent re-auth path was proven end-to-end: after deleting qa-e2e-admin's rows, the e2e admin spec ran against the stale storage state and passed with no login prompt (2 tests, 2.9 s).

Guard-chain flip recipe (adding a guarded vhost)

Move a public vhost from reverse_proxy <app>:<port> onto the chain: client → TLS edge → Anubis :8923 → outpost decision → internal http://:8017 listener → Host-matched handler → real upstream.

Order (never reorder):

1. Provider first. Create the outpost ProxyProvider before touching routes, or authed traffic hits the outpost error page instead of the app. Fleet convention for external_host is the exact browser-facing origin (the inventory table carries both bare origins and trailing slashes; routing matches on hostname, verified 2026-09-02). internal_host = http://caddy:8017, all three flows set, and an Application attached — the two provisioning rules above.
2. Route flip in ONE commit/deploy: the vhost becomes reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and inside the http://:8017 listener add @<svc> host <vhost> + handle @<svc> { reverse_proxy <upstream> }. Anubis guard set, Caddy TLS edge, :8017 routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).

Verification (all three):

• Unauthenticated probe from a client: curl -s https://<vhost>/ → HTTP 200 challenge page; grep the body for challenge|not a bot|anubis (~8 markers on a live challenge). A 502 here instead of the challenge = chain mis-wired.
• Post-auth routing proof: grep the LIVE config inside the container — podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile" — after the deploy prints its reload line; confirm the reload actually ran, don't trust exit 0.
• Outpost reachability: podman exec authentik-server getent hosts caddy → 10.89.1.8 caddy.dns.podman (2026-10-18) — that is Caddy's address on the dankclaw_default podman network, the alias internal_host dials. It is a netavark allocation, not a pinned address and not a compose artifact (the compose stack is gone); never hard-code it.

Pitfalls: probing http://localhost:8017/ from the host returns HTTP 000 (connection refused) — the post-auth listener lives in the caddy container's netns; probe inside it or grep the config. Non-browser clients cannot solve the JS challenge — before flipping, audit service-to-service traffic: ingestion/push paths must target internal addresses (loki:3100), never a public hostname of a guarded service. Exempt-by-construction categories stay unguarded: the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth — document the exemption in the vhost comment.

Driving the ORM (ak shell)

All authentik object work happens through the server container's ORM shell — no repo change, no deploy (these providers are NOT repo-managed; capture new provider+application pairs as a blueprint or setup script to avoid drift).

ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF

• The heredoc IS the program: ak shell reads all of stdin and execs it (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven one).
• Bootstrap prints JSON log lines, then a shell banner and an "NNN objects imported automatically…" line — pipe through sed -n '/objects imported/,$p' to drop the noise (unverified 2026-10-18: the exact banner text and the 229-object count — no ak shell was driven against the live auth service for this audit).
• Import paths (each wrong guess costs a failed exec): User, Group, Application, ApplicationEntitlement, PropertyMapping → authentik.core.models; ProxyProvider → authentik.providers.proxy.models (NOT authentik.outposts.models); LDAPProvider → authentik.providers.ldap.models; authentik.core.models.application does NOT exist as an import path; there is no authentik.policies.group module in this build. (unverified 2026-10-18: these module paths were not re-executed — importing outside a configured Django shell fails with ImproperlyConfigured)
• For live DB truth prefer the postgres container, and it works without any credential — the local socket is trusted, so a read-only audit is just podman exec -i authentik-postgres psql -U authentik -d authentik -At -f - with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify before guessing: 2026.x renamed both tables and columns, and the models are MTI all the way down — authentik_core_provider holds id, name and the three *_flow_id columns; the proxy table is authentik_providers_proxy_proxyprovider (doubled proxy) and its PK column really is oauth2provider_ptr_id (proxy providers inherit the OAuth2 table in 2026.8), the LDAP table is authentik_providers_ldap_ldapprovider with PK provider_ptr_id; authentik_core_application attaches to a provider through provider_id (its PK is policybindingmodel_ptr_id, not id), and authentik_outposts_outpost is keyed by uuid. So Provider.objects.filter(type=…) raising FieldError is the same story from the ORM side: type lives in the subclass tables. Never select secret columns (client_secret, key, password) into a log.

User provisioning playbook (user-level account)

There is no User.create_user classmethod, and is_superuser is NOT a constructor kwarg (it is a property derived from group membership) — passing it raises TypeError: User() got unexpected keyword arguments:

from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...")        # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
    u.groups.add(Group.objects.get(name=g))

path defaults to users (regular). The access map (re-checked against the live DB 2026-10-18): every one of the nine proxy providers carries the same authorization flow UUID, default-provider-authorization-implicit-consent (name "Authorize Application"), i.e. there are no per-app policy bindings to delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed downstream —

• homepage-admins → Caddy RBAC for homepage /admin (X-Authentik-Groups regex gate) and the homepage admin panel; regular users get only category: user panels from services.json.
• openwebui_users / openwebui_admins → ApplicationEntitlements "Open WebUI Users" and "Open WebUI Admins", both rows attached to the Open WebUI application (authentik_core_applicationentitlement, 2026-10-18), consumed by the "Open WebUI roles" property mapping → roles claim (unverified 2026-10-18: the group→entitlement binding and the mapping expression were not read; the mapping itself is not visible under that name in authentik_core_propertymapping). Verify live with u.app_entitlements(app).
• jellyfin_users / jellyfin_admins → Jellyfin via the LDAP outpost (filter (memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)).
• Non-Admin Users = the plain-user marker group (precedents: qa-e2e, and every human account).
• The human-user template (ridley, kassie, alex, artem, kirby, ry, nic, debo, freya, mitch, eric). The pks in this page are confirmed against authentik_core_user on 2026-10-18: ridley 13, kassie 14, alex 15, artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 — and every one carries the <user>@franksdank.cloud address the template prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked (unverified 2026-10-18). Template: username + <user>@franksdank.cloud + display name + is_active=True + the password set via set_password, membership in Non-Admin Users + jellyfin_users + openwebui_users. alex was recreated from scratch on 2026-10-08 into this shape (the old row had an empty email and no openwebui_users; audit first — the row carried no MFA device, token, entitlement, or proxy session, and events survive a user delete because Event.user is a denormalized JSON field, not a FK).
• EXCLUDE from a user-level account: homepage-admins, jellyfin_admins, openwebui_admins, authentik Admins, authentik Read-only.
• Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta) need no per-user group: any authenticated user passes the outpost.

Verify login behaviorally — authenticate() is the exact PasswordStage code path:

from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...")   # -> User
assert authenticate(request=None, username="ridley", password="nope") is None

Look flows up by slug, not name: Flow.objects.get(slug="default-authentication-flow") (its .name is "Welcome to authentik!" — .get(name=…) fails; both confirmed from authentik_flows_flow, 2026-10-18). Stage order, read from authentik_flows_flowstagebinding the same day: default-authentication-identification(10) → default-authentication-password(20) → default-authentication-mfa-validation(30; no-op with no enrolled device) → default-authentication-login(100). With no MFA enrolled, nothing else blocks.

More field gotchas (each cost a failed exec): ApplicationEntitlement.application can be null (guard with getattr); LDAPProvider has search_mode/bind_mode/base_dn and lacks search_base/of_groups/ organization — the backing table (authentik_providers_ldap_ldapprovider) has exactly provider_ptr_id, base_dn, certificate_id, tls_server_name, gid_start_number, uid_start_number, search_mode, bind_mode, mfa_support (information_schema.columns, 2026-10-18); OAuth2Provider has no .scopes (scopes are linked through ProviderPropertyMapping, which is why filter(provider=…) on a scope mapping is empty); Outpost.config is an OutpostConfig object, not a dict; Policy has no uniform .policy_cls (use type(p).__name__); User has no .status.

Never rotate AUTHENTIK_SECRET_KEY to fix claim issues — it invalidates every outpost session fleet-wide; purge the stale proxysession rows instead (section above).

Probing the outpost from inside the netns

Rootless podman gives each container its own netns; host ss -ltn cannot see listeners inside other containers' namespaces — a missing listener on the host is an observation artifact, not a fault. Distroless images have no shell (podman exec authentik-server sh fails), so probe by joining the namespace:

podman run --rm --network container:<container> --entrypoint /bin/busybox \
  docker.io/library/busybox:latest \
  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8

• Quote URLs over ssh: the remote shell is zsh and bare ? in query strings triggers globbing (no matches found).
• Boot transience: immediately after a restart the same probe can return 503 (configure task pending); poll until the answer changes before concluding.
• podman inspect --format "{{.Config.NetMode}}" errors on this podman version — the field is .HostConfig.NetworkMode.

Failure modes

Symptom	Cause	Fix
Every vhost 502s at the outpost	authentik-server down	podman logs authentik-server; check postgres health
Login loops (302 → login → 302)	trusted_proxies / X-Forwarded-* mismatch	Confirm Caddy sets X-Forwarded-For; the outpost trusts private_ranges
"Invalid signature" on the outpost	AUTHENTIK_SECRET_KEY differs between server and worker	Both services read the same .env var; re-deploy
Postgres "password authentication failed"	AUTHENTIK_PG_PASS changed after first init	Reset via pg_ctl or re-init the data dir
"Request has been denied: Invalid next URL" at sign-in	Flow entered with an absolute cross-host next= (old redirect design)	Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie
/outpost.goauthentik.io/* 404s on every host	Any provider row with a null flow (authentication/authorization/invalidation) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload	Backfill all three flows on every provider; watch podman logs authentik-server for Failed to fetch outpost from API
Login completes but lands on authentik's user UI (/if/user/…) instead of the app	Proxy provider has no attached Application (outpost cannot resolve the post-login redirect)	Create the Application with provider_id set — no error is logged, so test with a fresh browser context
Off-LAN clients get "hostname could not be found"	Guarded host missing from the public DNS zone	Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer
Correct membership but one user still 403s at a header gate	Outpost claim cache: a proxysession row minted before the membership change serves old claims until expiry	Delete that user's proxysession rows (section above) — next visit re-authorizes silently

Cross-references

• Caddy — the reverse proxy carrying the :8016/:8017 auth-chain listeners
• Pi-hole — the LAN DNS that resolves auth.franksdank.cloud
• Jellyfin — consumes the LDAP outpost via its LDAP-Auth plugin
• Network Architecture — the flat naming scheme
• Research: .docaudit/research/auth.md — the full provider comparison (the file is gone from the repo and from the host checkout as of 2026-10-18)