serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md

# Authentik — the homelab's SSO / OIDC provider

Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its embedded proxy outpost. Apps that
have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
Pi-hole) are not wrapped — they handle their own session.

- **Version:** `2026.8.0` (pinned in `dankclaw/authentik/quadlet/*.container`)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
  `authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)

## Why Authentik

The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
for all configuration, an outpost that terminates auth for reverse-proxied
apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
was removed in 2025.10, so the stack is 3 services, not 4. (The integration
mechanism changed in 2026.x — see Architecture; the research file predates
the pin.)

## Architecture

```mermaid
graph LR
    U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
    A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
    C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
    S -->|4. no session: login flow served in place| U
    S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
    P -->|6. app response back up the chain| U
    S --> PG[(postgres)]
```

Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:

- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
  exist in the 2026.8 URL tree — every unauthenticated subrequest got
  authentik's own 404 page.
- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
  in this build: a request carrying Caddy's full meta-header set still 404s,
  while the same request with only a matching `Host` answers. The embedded
  outpost routes **solely on the original `Host`**, matched against each
  attached proxy provider's `external_host`.
- Consequence: **every guarded host needs its own proxy provider** (exact
  `external_host`), and `:8016` is a single uniform `reverse_proxy` to
  `authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
  served in place; authenticated → forwarded to the provider's
  `internal_host` (`http://caddy:8017`), which routes straight to the app.
  The chain terminates there — nothing dials back into Authentik from
  `:8017`.

## The three compose services

| Service | Role | Image |
|---|---|---|
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.0` |
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.0` |

- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
  moved to PostgreSQL in 2025.10 (Redis removed).
- **Server** and **worker** run the same image; the `command` selects the role
  (`server` / `worker`).
- The worker runs as `0:0` (rootless: container root maps to the host user that
  owns the data dir).
- Data persists under `/home/dank/authentik/` (bind mounts, repo policy — no
  named volumes).

## Caddy integration

Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
proxies **every request whole** to the Authentik server with `Host`
preserved — there is no `forward_auth` and no per-vhost block:

```caddyfile
http://:8016 {
	reverse_proxy authentik-server:9000 {
		header_up X-Forwarded-Proto https
	}
}
```

The embedded outpost inside the server process decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.

**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.

Live provider inventory (provisioned via ORM, 2026-08-30), all attached to
outpost `7e39655b-…` ("authentik Embedded Outpost"), all with
`internal_host = http://caddy:8017` and all three flows set:

| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 3 | hermes-proxy | `https://hermes.franksdank.cloud` | hermes |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |

Provisioning pitfall (verified live): a ProxyProvider created with only
`authentication_flow` set — `authorization_flow`/`invalidation_flow` null —
makes the embedded outpost thread fail its own config fetch: the Rust side
logs `Failed to fetch outpost from API, retrying in 3 seconds` forever and
the entire `/outpost.goauthentik.io` space 404s for every host. Backfill all
three flows (pk1's values: `default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`) and the retry loop recovers on its next
3-second cycle.


## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)

The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
implementation diverges from upstream in ways that bit us; all verified live:

- **Application row required.** `OAuth2Provider.objects.create()` does not
  create the single-table-inheritance `Application` row; without it the flow
  executor can never complete an authorize request. Create both rows.
- **`grant_types` defaults to empty.** Until the provider's `grant_types`
  contains `"authorization_code"` (enum value with underscore), the token
  endpoint rejects every exchange with `invalid_request`.
- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
  `parse_token_request` see no `grant_type` → `unsupported_grant_type`;
  POST-body client secrets are not accepted — authenticate the confidential
  client with `Authorization: Basic base64(client_id:client_secret)`.
- **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
  `/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
  becomes the *expression source* instead of the context; every call raises a
  swallowed `PropertyMappingExpressionException` and the claim is skipped.
  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
  provider FK (linkage lives in the `ProviderPropertyMapping` through model),
  so `filter(provider=...)` always yields an empty queryset and tokens asked
  to carry scopes get zero. Net effect: userinfo/introspection answers with
  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
  build until the evaluator is patched or upgraded. Clients consuming this
  provider must key identity off `sub`.
- Scope mappings for this provider exist (`grafana-email-scope`,
  `grafana-profile-scope`, Python-syntax expressions) and will start emitting
  claims the moment the evaluator bug is fixed — no config change needed then.

## Configuration (secrets)

Two secrets live in the host `.env` (git-ignored; placeholders in
`.env.example`):

- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.

## First-time setup

1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
   health-gated), reloads Caddy.
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
   creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: **Directory →
   Applications** → add an application + a **Proxy** provider whose
   `external_host` is exactly the browser-facing origin
   (`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
   flows set (see the provisioning pitfall above). Attach the provider to the
   embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
   Connect** provider, create the app with the correct redirect URI, and paste
   the `client_id` / `client_secret` into the app's SSO settings.

## Operations

- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
  probes this).
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
  truth). The `/home/dank/authentik/` dir holds media + static files.
- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
  `./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.

## Failure modes

| Symptom | Cause | Fix |
|---|---|---|
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | A provider row with null `authorization_flow`/`invalidation_flow` wedges the outpost config fetch | Backfill the flows; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |

## Cross-references

- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison