serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

# Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its embedded proxy outpost. Apps that
have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
Pi-hole) are not wrapped — they handle their own session.

- **Version:** `2026.8.0` (pinned in `dankclaw/authentik/quadlet/*.container`)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
  `authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
- **LDAP outpost:** a 4th, quadlet-deployed unit (`authentik-outpost-ldap`)
  serves LDAP for Jellyfin — see the section below.

## Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
for all configuration, an outpost that terminates auth for reverse-proxied
apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
was removed in 2025.10, so the stack is 3 services, not 4. (The integration
mechanism changed in 2026.x — see Architecture; the research file predates
the pin.)

## Architecture

```mermaid
graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]
```

Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:

- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
  exist in the 2026.8 URL tree — every unauthenticated subrequest got
  authentik's own 404 page.
- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
  in this build: a request carrying Caddy's full meta-header set still 404s,
  while the same request with only a matching `Host` answers. The embedded
  outpost routes **solely on the original `Host`**, matched against each
  attached proxy provider's `external_host`.
- Consequence: **every guarded host needs its own proxy provider** (exact
  `external_host`), and `:8016` is a single uniform `reverse_proxy` to
  `authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
  served in place; authenticated → forwarded to the provider's
  `internal_host` (`http://caddy:8017`), which routes straight to the app.
  The chain terminates there — nothing dials back into Authentik from
  `:8017`.

## The three compose services

| Service | Role | Image |
|---|---|---|
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.0` |
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.0` |

- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
  moved to PostgreSQL in 2025.10 (Redis removed).
- **Server** and **worker** run the same image; the `command` selects the role
  (`server` / `worker`).
- The worker runs as `0:0` (rootless: container root maps to the host user that
  owns the data dir).
- Data persists under `/home/dank/Containers/authentik/{data,postgres}`
  (bind mounts, repo policy — no named volumes; moved from `/home/dank/authentik/`
  in the 2026-09-01 quadlet standardization).

## Caddy integration

Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
proxies **every request whole** to the Authentik server with `Host`
preserved — there is no `forward_auth` and no per-vhost block:

```caddyfile
http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}
```

The embedded outpost inside the server process decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.

**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.

Live provider inventory (verified 2026-09-02), all attached to outpost
`7e39655b-…` ("authentik Embedded Outpost"), all with
`internal_host = http://caddy:8017`, all three flows set, and each linked to
an Application (see the provisioning rules below):

| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |

Provider 3 (`hermes-proxy`) still exists but is dormant — its vhost was
decommissioned from Caddy on 2026-08-31. Provider 9 (`grafana`) and 12
(`Seerr OIDC`) are OAuth2 relying-party providers, not proxy providers; the
outpost does not deserialize them. Trailing slashes on `external_host` (the
runbook-recipe style) and bare origins both intercept correctly — the outpost
matches on hostname.

Two provisioning rules, each verified live by an outage:

1. **All three flows are required.** A ProxyProvider created via raw API/ORM
   with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
   null wedges the embedded outpost's config fetch: the Rust side rejects the
   whole config JSON (serde: `invalid type: null, expected a formatted UUID
   string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
   forever, and **everything behind the outpost dies at the next outpost
   reload** — including hosts whose providers were fine (verified 2026-09-02
   when three providers provisioned without any flows took down every guarded
   vhost after a server restart). Backfill all three flows (copy from pk 1:
   `default-authentication-flow`,
   `default-provider-authorization-implicit-consent` (the implicit-consent
   authorization flow, `default-invalidation-flow`) and the retry loop
   recovers on its next 3-second cycle. The silent killer: while the outpost
   process keeps its last good in-memory config, everything works — the
   outage only surfaces on restart/reload.
2. **Every proxy provider needs an attached Application.** The outpost
   resolves the post-login redirect through the provider's application; a
   provider with no app completes the login (authentik session established)
   but bounces to authentik's own user UI (`/if/user/#/library`) instead of
   the app. No error is logged anywhere — the SSO "works" and the app never
   appears. Create the Application with `provider_id` set (the UI does this
   automatically; raw API/ORM drives must do it explicitly).


## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
implementation diverges from upstream in ways that bit us; all verified live:

- **Application row required.** `OAuth2Provider.objects.create()` does not
  create the single-table-inheritance `Application` row; without it the flow
  executor can never complete an authorize request. Create both rows.
- **`grant_types` defaults to empty.** Until the provider's `grant_types`
  contains `"authorization_code"` (enum value with underscore), the token
  endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
  `parse_token_request` see no `grant_type` → `unsupported_grant_type`;
  POST-body client secrets are not accepted — authenticate the confidential
  client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
  `/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
  becomes the *expression source* instead of the context; every call raises a
  swallowed `PropertyMappingExpressionException` and the claim is skipped.
  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
  provider FK (linkage lives in the `ProviderPropertyMapping` through model),
  so `filter(provider=...)` always yields an empty queryset and tokens asked
  to carry scopes get zero. Net effect: userinfo/introspection answers with
  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
  build until the evaluator is patched or upgraded. Clients consuming this
  provider must key identity off `sub`.
- Scope mappings for this provider exist (`grafana-email-scope`,
  `grafana-profile-scope`, Python-syntax expressions) and will start emitting
  claims the moment the evaluator bug is fixed — no config change needed then.

## LDAP outpost (Jellyfin auth, live 2026-09-01)

A fourth unit, `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, serves LDAP for Jellyfin's LDAP-Auth
plugin. The image is pinned by its **amd64 single-platform digest**
(`ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4`):
podman 5.7.0 refuses the
`ghcr.io/goauthentik/ldap:2026.8.0` tag because the tag resolves to a
*nested* manifest list (index → per-platform manifest lists) —
`unexpectedly received a manifest list instead of a manifest for a single
image`. Resolve once with `podman manifest inspect <image> | jq -r
'.manifests[] | select(.platform.architecture=="amd64") | .digest'` and pin
that digest.

- Ports **389** (ldap) and **636** (ldaps) publish on the LAN; Jellyfin on
  Dankserver binds `192.168.0.143:389`. The outpost token travels in
  `AUTHENTIK_LDAP_OUTPOST_TOKEN` from the host `.env` (git-ignored).

### Authentik-side objects (provisioned via `ak shell` ORM)

| Object | Value |
|---|---|
| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud` |
| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` |
| Service account | `svc-jellyfin-ldap` — passwordless/unusable; authenticates by token |
| Bind token | intent **`verification`**, non-expiring — its key is the LDAP bind password |
| RBAC role | `svc-jellyfin-ldap-search` carrying object-level permission `authentik_providers_ldap.search_full_directory` on the provider |
| Groups | `jellyfin_users`, `jellyfin_admins` |
| Application | `jellyfin` (slug `jellyfin`) |

Hard-won rules, each verified live:

- **Token intent is load-bearing.** A bind token with `intent=app_password`
  is rejected by the flow's password stage with `Invalid password`;
  `intent=verification` passes. The outpost's direct binder executes the
  provider's authorization flow, and that password stage only accepts
  verification-intent tokens as static passwords.
- **Search needs `search_full_directory`.** Without the RBAC grant the
  outpost returns only the bind user itself — a `(memberOf=…)` filter then
  matches zero entries and the plugin logs `Found no users matching X`.
  The permission is object-level on the provider; grant it via an RBAC role
  and retest.
- `ak shell` import path for the provider model in 2026.8:
  `from authentik.providers.ldap.models import LDAPProvider`.

## Configuration (secrets)

Two secrets live in the host `.env` (git-ignored; placeholders in
`.env.example`):

- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.

## First-time setup

1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
   health-gated), reloads Caddy.
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
   creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: **Directory →
   Applications** → add an application + a **Proxy** provider whose
   `external_host` is exactly the browser-facing origin
   (`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
   flows set (see the provisioning pitfall above). Attach the provider to the
   embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
   Connect** provider, create the app with the correct redirect URI, and paste
   the `client_id` / `client_secret` into the app's SSO settings.

## Operations

- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
  probes this).
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
  truth). The `/home/dank/Containers/authentik/data` dir holds media + static files.
- **Upgrades:** bump the image tag in the three units under
  `dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik
  migrates the schema on startup.

## Outpost claim cache (`proxysession`) — 403 after a group add

The embedded outpost does not re-evaluate group membership per request. Each
completed authorize writes one row to `authentik_providers_proxy_proxysession`
whose `session_data` JSON holds the claims (`groups` included) that get served
for that browser session until the row expires (observed TTLs 1–24 h per
row). A session authorized **before** a user was added to a group keeps
serving the old group list — Caddy-side RBAC header gates then 403 despite
correct membership, and no authentik event is generated (the 403 happens at
Caddy; authentik never sees the request).

Diagnosis + fix (verified live 2026-09-01):

```sql
-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
       session_data->'claims'->>'groups'             AS cached_groups,
       expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';
```

The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s).

## Failure modes

| Symptom | Cause | Fix |
|---|---|---|
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | Any provider row with a null flow (`authentication`/`authorization`/`invalidation`) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload | Backfill all three flows on every provider; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Login completes but lands on authentik's user UI (`/if/user/…`) instead of the app | Proxy provider has no attached Application (outpost cannot resolve the post-login redirect) | Create the Application with `provider_id` set — no error is logged, so test with a fresh browser context |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |
| Correct membership but one user still 403s at a header gate | Outpost claim cache: a `proxysession` row minted before the membership change serves old claims until expiry | Delete that user's `proxysession` rows (section above) — next visit re-authorizes silently |

## Cross-references

- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison