serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

---
type: reference
description: "Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the"
tags:
  - infrastructure
  - authentik
timestamp: 2026-09-16
---

# Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its proxy outpost. Not wrapped: Gitea,
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
they handle their own session. Grafana is the exception that is both: it is a
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
`dankclaw/Caddyfile:374-377`; live provider table below).

- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
  the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
  2026-10-18)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack: five quadlet units, no compose.** `authentik-postgres`,
  `authentik-server`, `authentik-worker` (`deploy_authentik`,
  `scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
  (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
  `deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
  `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
  `scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
  `active`, `Restart=always`, `WantedBy=default.target`, with linger on
  (`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
  pre-quadlet era, retired.
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.

## Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
recorded in `.docaudit/research/auth.md`, which no longer exists in the repo or
in the host checkout (`ls`, 2026-10-18) *(unverified 2026-10-18: the
comparison itself)*. In short: full OIDC with single-logout, a web GUI for all
configuration, an outpost that terminates auth for reverse-proxied apps, and a
footprint that fits on a 3-user homelab — measured idle today
`podman stats --no-stream`: postgres 233.8 MB + server 739.1 MB + worker
292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ **1.6 GB** across five
containers (the older "~1–1.5 GB" figure was the three-service stack). Redis
was removed in 2025.10; the integration mechanism changed again in 2026.x (see
Architecture).

## Architecture

```mermaid
graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]
```

Who answers was re-measured 2026-10-18 from inside the `caddy` container with a
tenant-matching `Host: search.franksdank.cloud`:
`http://authentik-server:9000/outpost.goauthentik.io/start` → **302 into the
authentication flow**, while `http://authentik-outpost:9000/…` answers **404**
for `/outpost.goauthentik.io/*` *and* for `/if/flow/default-authentication-flow/`.
So the **server process** still terminates the proxy-provider decision, exactly
as Caddy's `:8016 → authentik-server:9000` hop assumes
(`dankclaw/Caddyfile:93-98`). The `authentik-outpost` unit is installed,
enabled (`generated`) and active, and it is the DB's `authentik Embedded
Outpost` (`7e39655b-…`, nine providers attached) holding the websocket channel
to core — but it serves no HTTP surface on the paths the chain needs. The unit
header's stronger claim that the URL space "is served by the Rust worker
binary, not the server"
(`dankclaw/authentik/quadlet/authentik-outpost.container:6-14`) is **not** what
this probe shows; do not point a new listener at `authentik-outpost:9000`
without re-measuring. Note also that a probe without a tenant `Host` (e.g.
`curl http://127.0.0.1:9000/…`) 404s on paths that work — that is the
Host-routing rule below, not an outage.

Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:

- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
  exist in the 2026.8 URL tree — every unauthenticated subrequest got
  authentik's own 404 page.
- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
  in this build: a request carrying Caddy's full meta-header set still 404s,
  while the same request with only a matching `Host` answers. The embedded
  outpost routes **solely on the original `Host`**, matched against each
  attached proxy provider's `external_host`.
- Consequence: **every guarded host needs its own proxy provider** (exact
  `external_host`), and `:8016` is a single uniform `reverse_proxy` to
  `authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
  served in place; authenticated → forwarded to the provider's
  `internal_host` (`http://caddy:8017`), which routes straight to the app.
  The chain terminates there — nothing dials back into Authentik from
  `:8017`.

## The three compose services

| Service | Role | Image |
|---|---|---|
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.3` |
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.3` |

- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
  moved to PostgreSQL in 2025.10 (Redis removed).
- **Server** and **worker** run the same image; the `command` selects the role
  (`server` / `worker`).
- The worker runs as `0:0` (rootless: container root maps to the host user that
  owns the data dir).
- Data persists under `/home/dank/Containers/authentik/{data,postgres}`
  (bind mounts, repo policy — no named volumes; moved from `/home/dank/authentik/`
  in the 2026-09-01 quadlet standardization).

## Caddy integration

Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
proxies **every request whole** to the Authentik server with `Host`
preserved — there is no `forward_auth` and no per-vhost block:

```caddyfile
http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}
```

The outpost decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.

**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.

Live provider inventory (read from the `authentik-postgres` container with
read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
members in `authentik_outposts_outpost_providers`), all with
`internal_host = http://caddy:8017`, all carrying the **same three flow
UUIDs** (`default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`), and each backed by an Application
(the `slug` column below):

| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |

That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
the lockstep rule holds today. Two corrections against the 2026-09-02 table:
**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
not proxy providers, so the proxy outpost does not deserialize them: pk 9
`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
below rejects), and pk 14 `Open WebUI`
(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
`external_host` (the runbook-recipe style) and bare origins both intercept
correctly — the outpost matches on hostname.

Two provisioning rules, each verified live by an outage:

1. **All three flows are required.** A ProxyProvider created via raw API/ORM
   with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
   null wedges the embedded outpost's config fetch: the Rust side rejects the
   whole config JSON (serde: `invalid type: null, expected a formatted UUID
   string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
   forever, and **everything behind the outpost dies at the next outpost
   reload** — including hosts whose providers were fine (verified 2026-09-02
   when three providers provisioned without any flows took down every guarded
   vhost after a server restart). Backfill all three flows (copy from pk 1:
   `default-authentication-flow`,
   `default-provider-authorization-implicit-consent` (the implicit-consent
   authorization flow, `default-invalidation-flow`) and the retry loop
   recovers on its next 3-second cycle. The silent killer: while the outpost
   process keeps its last good in-memory config, everything works — the
   outage only surfaces on restart/reload.
2. **Every proxy provider needs an attached Application.** The outpost
   resolves the post-login redirect through the provider's application; a
   provider with no app completes the login (authentik session established)
   but bounces to authentik's own user UI (`/if/user/#/library`) instead of
   the app. No error is logged anywhere — the SSO "works" and the app never
   appears. Create the Application with `provider_id` set (the UI does this
   automatically; raw API/ORM drives must do it explicitly).


## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
implementation diverges from upstream in ways that bit us; all verified live:

- **Application row required.** `OAuth2Provider.objects.create()` does not
  create the single-table-inheritance `Application` row; without it the flow
  executor can never complete an authorize request. Create both rows.
- **`grant_types` defaults to empty.** Until the provider's `grant_types`
  contains `"authorization_code"` (enum value with underscore), the token
  endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
  `parse_token_request` see no `grant_type` → `unsupported_grant_type`;
  POST-body client secrets are not accepted — authenticate the confidential
  client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
  `/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
  becomes the *expression source* instead of the context; every call raises a
  swallowed `PropertyMappingExpressionException` and the claim is skipped.
  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
  provider FK (linkage lives in the `ProviderPropertyMapping` through model),
  so `filter(provider=...)` always yields an empty queryset and tokens asked
  to carry scopes get zero. Net effect: userinfo/introspection answers with
  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
  build until the evaluator is patched or upgraded. Clients consuming this
  provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
  the evaluator bug is fixed; no config change is needed then.

## LDAP outpost (Jellyfin auth, live 2026-09-01)

The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
single-platform digest**
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
(`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26`, matching the
running image in `podman ps`, 2026-10-18). The digest in this page before 2026-10-18
(`…fa33bc8f…`) is superseded. The reason for pinning a digest at all: podman
5.7.0 refuses the plain tag because it resolves to a *nested* manifest list
(index → per-platform manifest lists) — `unexpectedly received a manifest list
instead of a manifest for a single image` — first hit on `:2026.8.0`, still the
rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
<image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") |
.digest'` and pin that digest.

- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
  (`PublishPort=389:3389`, `636:6636`,
  `dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
  shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
  them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
  (2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
  from the host `dankclaw/.env` (git-ignored).

### Authentik-side objects (provisioned via `ak shell` ORM)

Read from the live database (read-only `SELECT`s against the
`authentik-postgres` container, 2026-10-18):

| Object | Value |
|---|---|
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` |
| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) |

Hard-won rules (the first is now contested by the live database):

- **Token intent: contested, do not trust either value blindly.** This page
  long asserted that a bind token with `intent=app_password` is rejected by the
  flow's password stage with `Invalid password` and only `intent=verification`
  passes. The live token for `svc-jellyfin-ldap` is `app_password`
  (`authentik_core_token`, 2026-10-18), so either the rule is wrong or the
  current Jellyfin bind path is not that token. Nothing observable settles it
  today: the LDAP outpost's retained log holds only its 5-minute
  `initialized direct binder` / `Update providers` config cycles and no bind
  activity *(unverified 2026-10-18: no bind was performed — reproducing it
  requires the bind secret)*. Re-verify with a real bind before changing the
  token's intent either way.
- **Search needs `search_full_directory`.** Without the RBAC grant the
  outpost returns only the bind user itself — a `(memberOf=…)` filter then
  matches zero entries and the plugin logs `Found no users matching X`.
  The permission is object-level on the provider; grant it via an RBAC role
  and retest.
- `ak shell` import path for the provider model in 2026.8:
  `from authentik.providers.ldap.models import LDAPProvider`.
  *(unverified 2026-10-18: importing it outside a configured Django shell fails
  with `ImproperlyConfigured`, and no `ak shell` was driven against the live
  auth service for this check)*

## Configuration (secrets)

**Four** variables live in the host env file the units read,
`/home/dank/serverAppConfigs/dankclaw/.env` (git-ignored; the quadlets expand
`${…}` from it via `EnvironmentFile`):

- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
- `AUTHENTIK_OUTPOST_TOKEN` — proxy outpost token
  (`dankclaw/authentik/quadlet/authentik-outpost.container:34`).
- `AUTHENTIK_LDAP_OUTPOST_TOKEN` — LDAP outpost token
  (`…/authentik-outpost-ldap.container:29`).

Only the first two have placeholders in `dankclaw/.env.example`; the two outpost
tokens have no example entry (there is no repository-root `.env.example` —
`find`, 2026-10-18). Rotating `AUTHENTIK_SECRET_KEY` fleet-wide invalidates
every outpost session (see the ORM playbook).

## First-time setup

1. `./scripts/deploy.sh dankclaw --service authentik-server` (any of
   `authentik-postgres|authentik-server|authentik-worker` runs
   `deploy_authentik`, `scripts/deploy-remote.sh:980-1004`) — installs the three
   core quadlets, starts postgres health-gated (`wait_container_healthy
   authentik-postgres 120`), then the server (300 s health wait) and the worker.
   The outposts are separate handlers: `--service authentik-outpost` and
   `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
   dispatch at `:1579-1588`).
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
   creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: **Directory →
   Applications** → add an application + a **Proxy** provider whose
   `external_host` is exactly the browser-facing origin
   (`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
   flows set (see the provisioning pitfall above). Attach the provider to the
   embedded outpost.
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
   the app with the correct redirect URI, and paste the `client_id` /
   `client_secret` into the app's SSO settings. The relying parties that exist
   today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
   **Gitea has no provider row at all** (live provider table, 2026-10-18) — it
   authenticates itself and stays off the chain by design
   (`dankclaw/Caddyfile:110-120`).

## Operations

- **Health:** there is **no** HTTP health path on this build —
  `/-/health/`, `/-/live/` and `/api/-/health/` all answer **404**, on
  `127.0.0.1:9000` and through `https://auth.franksdank.cloud` alike
  (`curl`, 2026-10-18). The container's own probe is Django-side:
  `HealthCmd=python /manage.py check --database default`
  (`dankclaw/authentik/quadlet/authentik-server.container:36`), and
  `podman inspect authentik-server` reports `healthy`. A usable liveness URL is
  a flow page: `https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`
  → 200 (2026-10-18). The earlier "the server healthcheck probes `/-/health/`"
  line was wrong in both halves.
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
  truth). The `/home/dank/Containers/authentik/data` dir holds media + static
  files (mounted by all three app units).
- **Upgrades:** bump the tag in **all three app units** under
  `dankclaw/authentik/quadlet/` and re-deploy; `authentik-outpost` carries the
  same tag (it runs the same image, `Exec=worker`) and `authentik-outpost-ldap`
  is **digest-pinned**, so its pin must be re-resolved for the new version (see
  the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
  Authentik migrates the schema on startup.

## Self-serve user-settings flow (password change)

Users change their own account details through the `default-user-settings-flow`
(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the
target of the homepage footer's "account" link. It is a `stage_configuration`
designated flow, so it renders every bound `ConfigurableStage` for the logged-in
user.

Stages, in execution order (the `FlowStageBinding.order` field, not the M2M
insertion order):

| order | stage | type | what it does |
|---|---|---|---|
| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** |
| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user |

**Password change is a prompt field, not a stage.** The `UserWriteStage`
writes whatever the preceding `PromptStage` collected: for the key `password`
it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix
is a `password` field on the `default-user-settings` prompt
(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a
`PasswordStage`. A `PasswordStage` is an *authentication* stage: its
`validate_password` runs `authenticate()` against the backends to check the
*current* password; it never sets a new one. (A `PasswordStage` was added on
2026-09-25 and removed the same day for exactly this reason.)

Two ORM gotchas: `Flow.stages.add()` fails on this through model
(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
MTI), so create the `FlowStageBinding` row directly with an explicit `order`;
and a prompt field is a `Prompt` row linked through `PromptStageField`
(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).

**A blank password must not set an empty password.** `set_password("")` would
make the account log in with no password, so a validation policy
(`default-user-settings-no-empty-password`, bound to the prompt via
`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
is `""` and allows it when set. Verified with the `PolicyEvaluator`
(empty → deny, set → allow, absent → allow) and end-to-end by driving
`UserWriteStageView.update_user` on a throwaway user (new password valid, old
invalid).

## Outpost claim cache (`proxysession`) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each
completed authorize writes one row to `authentik_providers_proxy_proxysession`
whose `session_data` JSON holds the claims (`groups` included) that get served
for that browser session until the row expires (observed TTLs 1–24 h per
row). A session authorized **before** a user was added to a group keeps
serving the old group list — Caddy-side RBAC header gates then 403 despite
correct membership, and no authentik event is generated (the 403 happens at
Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

```sql
-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';
```

The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s).

## Guard-chain flip recipe (adding a guarded vhost)

Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
client → TLS edge → Anubis :8923 → outpost decision → internal
`http://:8017` listener → Host-matched handler → real upstream.

**Order (never reorder):**

1. **Provider first.** Create the outpost ProxyProvider before touching
   routes, or authed traffic hits the outpost error page instead of the app.
   Fleet convention for `external_host` is the exact browser-facing origin
   (the inventory table carries both bare origins and trailing slashes;
   routing matches on hostname, verified 2026-09-02).
   `internal_host = http://caddy:8017`, all three flows set, and an
   Application attached — the two provisioning rules above.
2. **Route flip in ONE commit/deploy**: the vhost becomes
   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
   inside the `http://:8017` listener add `@<svc> host <vhost>` +
   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
   edge, `:8017` routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).

**Verification (all three):**

- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
  challenge page; grep the body for `challenge|not a bot|anubis` (~8
  markers on a live challenge). A 502 here instead of the challenge = chain
  mis-wired.
- Post-auth routing proof: grep the LIVE config inside the container —
  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
  after the deploy prints its reload line; confirm the reload actually ran,
  don't trust exit 0.
- Outpost reachability: `podman exec authentik-server getent hosts caddy` →
  `10.89.1.8 caddy.dns.podman` (2026-10-18) — that is Caddy's address on the
  `dankclaw_default` podman network, the alias `internal_host` dials. It is a
  netavark allocation, not a pinned address and not a compose artifact (the
  compose stack is gone); never hard-code it.

**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
000 (connection refused) — the post-auth listener lives in the caddy
container's netns; probe inside it or grep the config. Non-browser clients
cannot solve the JS challenge — before flipping, audit service-to-service
traffic: ingestion/push paths must target internal addresses (`loki:3100`),
never a public hostname of a guarded service. Exempt-by-construction
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
endpoints, services with their own complete auth — document the exemption
in the vhost comment.

## Driving the ORM (`ak shell`)

All authentik object work happens through the server container's ORM shell —
no repo change, no deploy (these providers are NOT repo-managed; capture new
provider+application pairs as a blueprint or setup script to avoid drift).

```bash
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF
```

- The heredoc IS the program: `ak shell` reads all of stdin and execs it
  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
  one).
- Bootstrap prints JSON log lines, then a shell banner and an
  "`NNN objects imported automatically…`" line — pipe through
  `sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
  the exact banner text and the 229-object count — no `ak shell` was driven
  against the live auth service for this audit)*.
- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
  `Application`, `ApplicationEntitlement`, `PropertyMapping` →
  `authentik.core.models`; `ProxyProvider` →
  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
  `LDAPProvider` → `authentik.providers.ldap.models`;
  `authentik.core.models.application` does NOT exist as an import path; there
  is no `authentik.policies.group` module in this build.
  *(unverified 2026-10-18: these module paths were not re-executed — importing
  outside a configured Django shell fails with `ImproperlyConfigured`)*
- For live DB truth prefer the postgres container, and it works without any
  credential — the local socket is trusted, so a read-only audit is just
  `podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
  with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
  before guessing: 2026.x renamed both tables and columns, and the models are
  MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
  three `*_flow_id` columns; the proxy table is
  `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
  really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
  table in 2026.8), the LDAP table is
  `authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
  `authentik_core_application` attaches to a provider through `provider_id`
  (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
  is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
  is the same story from the ORM side: `type` lives in the subclass tables.
  Never select secret columns (`client_secret`, `key`, `password`) into a log.

## User provisioning playbook (user-level account)

There is no `User.create_user` classmethod, and `is_superuser` is NOT a
constructor kwarg (it is a property derived from group membership) — passing
it raises `TypeError: User() got unexpected keyword arguments`:

```python
from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...")        # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
    u.groups.add(Group.objects.get(name=g))
```

`path` defaults to `users` (regular). The access map (re-checked against the
live DB 2026-10-18): **every one of the nine proxy providers carries the same
authorization flow UUID**, `default-provider-authorization-implicit-consent`
(name "Authorize Application"), i.e. there are no per-app policy bindings to
delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed
downstream —

- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
  regex gate) and the homepage admin panel; regular users get only
  `category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
  **"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
  `Open WebUI` application (`authentik_core_applicationentitlement`,
  2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
  claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
  expression were not read; the mapping itself is not visible under that name
  in `authentik_core_propertymapping`)*. Verify live with
  `u.app_entitlements(app)`.
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
  every human account).
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
  debo, freya, mitch, eric). The pks in this page are confirmed against
  `authentik_core_user` on 2026-10-18: ridley 13, kassie 14, alex 15,
  artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
  and every one carries the `<user>@franksdank.cloud` address the template
  prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
  *(unverified 2026-10-18)*. Template: `username` +
  `<user>@franksdank.cloud` + display name + `is_active=True` + the password
  set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
  `openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
  shape (the old row had an empty email and no `openwebui_users`; audit first —
  the row carried no MFA device, token, entitlement, or proxy session, and
  events survive a user delete because `Event.user` is a denormalized JSON
  field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
  Plex, Seerr, Marreta) need no per-user group: any authenticated user
  passes the outpost.

Verify login behaviorally — `authenticate()` is the exact PasswordStage code
path:

```python
from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...")   # -> User
assert authenticate(request=None, username="ridley", password="nope") is None
```

Look flows up by slug, not name:
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
"Welcome to authentik!" — `.get(name=…)` fails; both confirmed from
`authentik_flows_flow`, 2026-10-18). Stage order, read from
`authentik_flows_flowstagebinding` the same day:
`default-authentication-identification`(10) →
`default-authentication-password`(20) →
`default-authentication-mfa-validation`(30; no-op with no enrolled device) →
`default-authentication-login`(100). With no MFA enrolled, nothing else blocks.

More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
`organization` — the backing table
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
(scopes are linked through `ProviderPropertyMapping`, which is why
`filter(provider=…)` on a scope mapping is empty); `Outpost.config` is an
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
(use `type(p).__name__`); `User` has no `.status`.

Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
every outpost session fleet-wide; purge the stale `proxysession` rows
instead (section above).

## Probing the outpost from inside the netns

Rootless podman gives each container its own netns; **host `ss -ltn` cannot
see listeners inside other containers' namespaces** — a missing listener on
the host is an observation artifact, not a fault. Distroless images have no
shell (`podman exec authentik-server sh` fails), so probe by joining the
namespace:

```
podman run --rm --network container:<container> --entrypoint /bin/busybox \
  docker.io/library/busybox:latest \
  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
```

- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
  triggers globbing (`no matches found`).
- Boot transience: immediately after a restart the same probe can return 503
  (configure task pending); poll until the answer changes before concluding.
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
  version — the field is `.HostConfig.NetworkMode`.

## Failure modes

| Symptom | Cause | Fix |
|---|---|---|
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | Any provider row with a null flow (`authentication`/`authorization`/`invalidation`) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload | Backfill all three flows on every provider; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Login completes but lands on authentik's user UI (`/if/user/…`) instead of the app | Proxy provider has no attached Application (outpost cannot resolve the post-login redirect) | Create the Application with `provider_id` set — no error is logged, so test with a fresh browser context |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |
| Correct membership but one user still 403s at a header gate | Outpost claim cache: a `proxysession` row minted before the membership change serves old claims until expiry | Delete that user's `proxysession` rows (section above) — next visit re-authorizes silently |

## Cross-references

- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison
  *(the file is gone from the repo and from the host checkout as of
  2026-10-18)*