Files
serverAppConfigs/serverAppConfigs-docs/Infrastructure/Authentik.md
Frank 327e78b950 docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
2026-10-19 10:56:37 -07:00

489 lines
27 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: reference
description: "Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the"
tags:
- infrastructure
- authentik
timestamp: 2026-09-16
---
# Authentik — the homelab's SSO / OIDC provider
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its proxy outpost. Not wrapped: Gitea,
OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
they handle their own session. Grafana is the exception that is both: it is a
guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`
`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]).
- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`
the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`
2026-10-18)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack: five quadlet units, no compose.** `authentik-postgres`
`authentik-server`, `authentik-worker` (`deploy_authentik`
`scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
(the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`
`deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
`authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`
`scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
`active`, `Restart=always`, `WantedBy=default.target`, with linger on
(`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
pre-quadlet era, retired.
- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
**Split off from this page:** [[Authentik Processes]] owns the per-container
facts for `authentik-postgres`, `authentik-server` and `authentik-worker`
[[Authentik Outpost Reference]] owns the provider/flow/token lookup material —
the live proxy-provider inventory, the proxy-provider provisioning rules, the
OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow
design, and the live database shape.
## Why Authentik
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
recorded in `.docaudit/research/auth.md`, which no longer exists in the repo or
in the host checkout (`ls`, 2026-10-18) *(unverified 2026-10-18: the
comparison itself)*. In short: full OIDC with single-logout, a web GUI for all
configuration, an outpost that terminates auth for reverse-proxied apps, and a
footprint that fits on a 3-user homelab — measured idle today
`podman stats --no-stream`: postgres 233.8 MB + server 739.1 MB + worker
292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ **1.6 GB** across five
containers (the older "~1–1.5 GB" figure was the three-service stack). Redis
was removed in 2025.10; the integration mechanism changed again in 2026.x (see
Architecture).
## Architecture
```mermaid
graph LR
U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
S -->|4. no session: login flow served in place| U
S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
P -->|6. app response back up the chain| U
S --> PG[(postgres)]
```
Who answers was re-measured 2026-10-18 from inside the `caddy` container with a
tenant-matching `Host: search.franksdank.cloud`:
`http://authentik-server:9000/outpost.goauthentik.io/start` → **302 into the
authentication flow**, while `http://authentik-outpost:9000/…` answers **404**
for `/outpost.goauthentik.io/*` *and* for `/if/flow/default-authentication-flow/`.
So the **server process** still terminates the proxy-provider decision, exactly
as Caddy's `:8016 → authentik-server:9000` hop assumes
(`dankclaw/Caddyfile:93-98`). The `authentik-outpost` unit is installed,
enabled (`generated`) and active, and it is the DB's `authentik Embedded
Outpost` (`7e39655b-…`, nine providers attached) holding the websocket channel
to core — but it serves no HTTP surface on the paths the chain needs. The unit
header's stronger claim that the URL space "is served by the Rust worker
binary, not the server"
(`dankclaw/authentik/quadlet/authentik-outpost.container:6-14`) is **not** what
this probe shows; do not point a new listener at `authentik-outpost:9000`
without re-measuring. Note also that a probe without a tenant `Host` (e.g.
`curl http://127.0.0.1:9000/…`) 404s on paths that work — that is the
Host-routing rule below, not an outage.
Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:
- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
exist in the 2026.8 URL tree — every unauthenticated subrequest got
authentik's own 404 page.
- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
in this build: a request carrying Caddy's full meta-header set still 404s,
while the same request with only a matching `Host` answers. The embedded
outpost routes **solely on the original `Host`**, matched against each
attached proxy provider's `external_host`.
- Consequence: **every guarded host needs its own proxy provider** (exact
`external_host`), and `:8016` is a single uniform `reverse_proxy` to
`authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
served in place; authenticated → forwarded to the provider's
`internal_host` (`http://caddy:8017`), which routes straight to the app.
The chain terminates there — nothing dials back into Authentik from
`:8017`.
## The stack's processes
Per-container facts — unit path, image and role, port, run-as, data directory,
health probe — are owned by [[Authentik Processes]].
## Caddy integration
Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
proxies **every request whole** to the Authentik server with `Host`
preserved — there is no `forward_auth` and no per-vhost block:
```caddyfile
http://:8016 {
reverse_proxy authentik-server:9000 {
header_up X-Forwarded-Proto https
}
}
```
The outpost decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.
**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
The live proxy-provider inventory (the per-vhost `external_host` → app mapping)
the two provisioning rules the outpost enforces, and this fork's OAuth2
relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
## LDAP outpost (Jellyfin auth, live 2026-09-01)
The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
→ `authentik-outpost-ldap.service`, one of the five units in this stack —
serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
single-platform digest**
`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
(`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26`, matching the
running image in `podman ps`, 2026-10-18). The digest in this page before 2026-10-18
(`…fa33bc8f…`) is superseded. The reason for pinning a digest at all: podman
5.7.0 refuses the plain tag because it resolves to a *nested* manifest list
(index → per-platform manifest lists) — `unexpectedly received a manifest list
instead of a manifest for a single image` — first hit on `:2026.8.0`, still the
rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
<image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") |
.digest'` and pin that digest.
- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
(`PublishPort=389:3389`, `636:6636`
`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
(2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
from the host `dankclaw/.env` (git-ignored).
The objects behind it — LDAP provider, bind DN, service account and bind token,
RBAC grant, groups, application slug — and the hard-won bind rules are in
[[Authentik Outpost Reference]].
## Configuration (secrets)
**Four** variables live in the host env file the units read,
`/home/dank/serverAppConfigs/dankclaw/.env` (git-ignored; the quadlets expand
`${…}` from it via `EnvironmentFile`):
- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
- `AUTHENTIK_OUTPOST_TOKEN` — proxy outpost token
(`dankclaw/authentik/quadlet/authentik-outpost.container:34`).
- `AUTHENTIK_LDAP_OUTPOST_TOKEN` — LDAP outpost token
(`…/authentik-outpost-ldap.container:29`).
Only the first two have placeholders in `dankclaw/.env.example`; the two outpost
tokens have no example entry (there is no repository-root `.env.example` —
`find`, 2026-10-18). Rotating `AUTHENTIK_SECRET_KEY` fleet-wide invalidates
every outpost session (see the ORM playbook).
## First-time setup
1. `./scripts/deploy.sh dankclaw --service authentik-server` (any of
`authentik-postgres|authentik-server|authentik-worker` runs
`deploy_authentik`, `scripts/deploy-remote.sh:980-1004`) — installs the three
core quadlets, starts postgres health-gated (`wait_container_healthy
authentik-postgres 120`), then the server (300 s health wait) and the worker.
The outposts are separate handlers: `--service authentik-outpost` and
`--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`
dispatch at `:1579-1588`).
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema.
3. For every guarded vhost, in the admin interface: **Directory →
Applications** → add an application + a **Proxy** provider whose
`external_host` is exactly the browser-facing origin
(`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
flows set (see the provisioning rules in
[[Authentik Outpost Reference]]). Attach the provider to the
embedded outpost.
4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
the app with the correct redirect URI, and paste the `client_id` /
`client_secret` into the app's SSO settings. The relying parties that exist
today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14)
**Gitea has no provider row at all** (live provider table, 2026-10-18) — it
authenticates itself and stays off the chain by design
(`dankclaw/Caddyfile:110-120`).
## Operations
- **Health:** there is **no** HTTP health path on this build —
`/-/health/`, `/-/live/` and `/api/-/health/` all answer **404**, on
`127.0.0.1:9000` and through `https://auth.franksdank.cloud` alike
(`curl`, 2026-10-18). The container's own probe is Django-side:
`HealthCmd=python /manage.py check --database default`
(`dankclaw/authentik/quadlet/authentik-server.container:36`), and
`podman inspect authentik-server` reports `healthy`. A usable liveness URL is
a flow page: `https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`
→ 200 (2026-10-18). The earlier "the server healthcheck probes `/-/health/`"
line was wrong in both halves.
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
truth). The `/home/dank/Containers/authentik/data` dir holds media + static
files (mounted by all three app units).
- **Upgrades:** bump the tag in **all three app units** under
`dankclaw/authentik/quadlet/` and re-deploy; `authentik-outpost` carries the
same tag (it runs the same image, `Exec=worker`) and `authentik-outpost-ldap`
is **digest-pinned**, so its pin must be re-resolved for the new version (see
the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
Authentik migrates the schema on startup.
## Outpost claim cache (`proxysession`) — 403 after a group add
The embedded outpost does not re-evaluate group membership per request. Each
completed authorize writes one row to `authentik_providers_proxy_proxysession`
whose `session_data` JSON holds the claims (`groups` included) that get served
for that browser session until the row expires (observed TTLs 1–24 h
row). A session authorized **before** a user was added to a group keeps
serving the old group list — Caddy-side RBAC header gates then 403 despite
correct membership, and no authentik event is generated (the 403 happens at
Caddy; authentik never sees the request).
Diagnosis + fix (verified live 2026-09-01):
```sql
-- who holds which cached groups (run in the authentik-postgres container)
SELECT session_data->'claims'->>'preferred_username' AS user,
session_data->'claims'->>'groups' AS cached_groups,
expires
FROM authentik_providers_proxy_proxysession ORDER BY expires;
-- invalidate one user's cache; their authentik login session is untouched,
-- so the next visit re-authorizes silently and mints fresh claims
DELETE FROM authentik_providers_proxy_proxysession
WHERE session_data->'claims'->>'preferred_username' = 'akadmin';
```
The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s).
## Guard-chain flip recipe (adding a guarded vhost)
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
client → TLS edge → Anubis :8923 → outpost decision → internal
`http://:8017` listener → Host-matched handler → real upstream.
**Order (never reorder):**
1. **Provider first.** Create the outpost ProxyProvider before touching
routes, or authed traffic hits the outpost error page instead of the app.
Fleet convention for `external_host` is the exact browser-facing origin
(the inventory table carries both bare origins and trailing slashes;
routing matches on hostname, verified 2026-09-02).
`internal_host = http://caddy:8017`, all three flows set, and an
Application attached — the two provisioning rules in [[Authentik Outpost Reference]].
2. **Route flip in ONE commit/deploy**: the vhost becomes
`reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
inside the `http://:8017` listener add `@<svc> host <vhost>` +
`handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
edge, `:8017` routing table, and provider list change together.
3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
**Verification (all three):**
- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
challenge page; grep the body for `challenge|not a bot|anubis` (~8
markers on a live challenge). A 502 here instead of the challenge = chain
mis-wired.
- Post-auth routing proof: grep the LIVE config inside the container —
`podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
after the deploy prints its reload line; confirm the reload actually ran,
don't trust exit 0.
- Outpost reachability: `podman exec authentik-server getent hosts caddy` →
`10.89.1.8 caddy.dns.podman` (2026-10-18) — that is Caddy's address on the
`dankclaw_default` podman network, the alias `internal_host` dials. It is a
netavark allocation, not a pinned address and not a compose artifact (the
compose stack is gone); never hard-code it.
**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
000 (connection refused) — the post-auth listener lives in the caddy
container's netns; probe inside it or grep the config. Non-browser clients
cannot solve the JS challenge — before flipping, audit service-to-service
traffic: ingestion/push paths must target internal addresses (`loki:3100`)
never a public hostname of a guarded service. Exempt-by-construction
categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
endpoints, services with their own complete auth — document the exemption
in the vhost comment.
## Driving the ORM (`ak shell`)
All authentik object work happens through the server container's ORM shell —
no repo change, no deploy (these providers are NOT repo-managed; capture new
provider+application pairs as a blueprint or setup script to avoid drift).
```bash
ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
from authentik.core.models import User, Group
...
EOF
```
- The heredoc IS the program: `ak shell` reads all of stdin and execs it
(heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
one).
- Bootstrap prints JSON log lines, then a shell banner and an
"`NNN objects imported automatically…`" line — pipe through
`sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
the exact banner text and the 229-object count — no `ak shell` was driven
against the live auth service for this audit)*.
- Import paths (each wrong guess costs a failed exec): `User`, `Group`
`Application`, `ApplicationEntitlement`, `PropertyMapping` →
`authentik.core.models`; `ProxyProvider` →
`authentik.providers.proxy.models` (NOT `authentik.outposts.models`)
`LDAPProvider` → `authentik.providers.ldap.models`
`authentik.core.models.application` does NOT exist as an import path; there
is no `authentik.policies.group` module in this build.
- (unverified 2026-10-18: these module paths were not re-executed — importing
outside a configured Django shell fails with `ImproperlyConfigured`)*
- The live database's table and column names (the 2026.x renames, the MTI parent
keys, the read-only `psql` audit recipe) are reference lookup:
[[Authentik Outpost Reference]].
## User provisioning playbook (user-level account)
There is no `User.create_user` classmethod, and `is_superuser` is NOT a
constructor kwarg (it is a property derived from group membership) — passing
it raises `TypeError: User() got unexpected keyword arguments`:
```python
from authentik.core.models import User, Group
u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
u.set_password("...") # password arrives on stdin only, never argv
u.save()
for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
u.groups.add(Group.objects.get(name=g))
```
`path` defaults to `users` (regular). The access map (re-checked against the
live DB 2026-10-18): **every one of the nine proxy providers carries the same
authorization flow UUID**, `default-provider-authorization-implicit-consent`
(name "Authorize Application"), i.e. there are no per-app policy bindings to
delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed
downstream —
- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
regex gate) and the homepage admin panel; regular users get only
`category: user` panels from `services.json`.
- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
**"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
`Open WebUI` application (`authentik_core_applicationentitlement`
2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
expression were not read; the mapping itself is not visible under that name
in `authentik_core_propertymapping`)*. Verify live with
`u.app_entitlements(app)`.
- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
(filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
every human account).
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
debo, freya, mitch, eric). The pks in this page are confirmed against
`authentik_core_user` on 2026-10-18: ridley 13, kassie 14, alex 15,
artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
and every one carries the `<user>@franksdank.cloud` address the template
prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
- (unverified 2026-10-18)*. Template: `username` +
`<user>@franksdank.cloud` + display name + `is_active=True` + the password
set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
`openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
shape (the old row had an empty email and no `openwebui_users`; audit first —
the row carried no MFA device, token, entitlement, or proxy session, and
events survive a user delete because `Event.user` is a denormalized JSON
field, not a FK).
- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`
`openwebui_admins`, `authentik Admins`, `authentik Read-only`.
- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
Plex, Seerr, Marreta) need no per-user group: any authenticated user
passes the outpost.
Verify login behaviorally — `authenticate()` is the exact PasswordStage code
path:
```python
from django.contrib.auth import authenticate
assert authenticate(request=None, username="ridley", password="...") # -> User
assert authenticate(request=None, username="ridley", password="nope") is None
```
Look flows up by slug, not name:
`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
"Welcome to authentik!" — `.get(name=…)` fails; both confirmed from
`authentik_flows_flow`, 2026-10-18). Stage order, read from
`authentik_flows_flowstagebinding` the same day:
`default-authentication-identification`(10) →
`default-authentication-password`(20) →
`default-authentication-mfa-validation`(30; no-op with no enrolled device) →
`default-authentication-login`(100). With no MFA enrolled, nothing else blocks.
More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
can be null (guard with `getattr`); `LDAPProvider` has
`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
`organization` — the backing table
(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`
`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`
`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
(scopes are linked through `ProviderPropertyMapping`, which is why
`filter(provider=…)` on a scope mapping is empty); `Outpost.config` is an
`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
(use `type(p).__name__`); `User` has no `.status`.
Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
every outpost session fleet-wide; purge the stale `proxysession` rows
instead (section above).
## Probing the outpost from inside the netns
Rootless podman gives each container its own netns; **host `ss -ltn` cannot
see listeners inside other containers' namespaces** — a missing listener on
the host is an observation artifact, not a fault. Distroless images have no
shell (`podman exec authentik-server sh` fails), so probe by joining the
namespace:
```
podman run --rm --network container:<container> --entrypoint /bin/busybox \
docker.io/library/busybox:latest \
wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
```
- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
triggers globbing (`no matches found`).
- Boot transience: immediately after a restart the same probe can return 503
(configure task pending); poll until the answer changes before concluding.
- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
version — the field is `.HostConfig.NetworkMode`.
## Failure modes
| Symptom | Cause | Fix |
|---|---|---|
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | Any provider row with a null flow (`authentication`/`authorization`/`invalidation`) wedges the outpost config fetch — all guarded vhosts die at the next outpost reload | Backfill all three flows on every provider; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Login completes but lands on authentik's user UI (`/if/user/…`) instead of the app | Proxy provider has no attached Application (outpost cannot resolve the post-login redirect) | Create the Application with `provider_id` set — no error is logged, so test with a fresh browser context |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |
| Correct membership but one user still 403s at a header gate | Outpost claim cache: a `proxysession` row minted before the membership change serves old claims until expiry | Delete that user's `proxysession` rows (section above) — next visit re-authorizes silently |
## Cross-references
- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison
- (the file is gone from the repo and from the host checkout as of
2026-10-18)*