From ee1e038f302adfa17ef2f97c6fdfecb9e4c210b6 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 29 Aug 2026 23:55:48 -0700 Subject: [PATCH] docs: authentik/anubis/network corrected to live-verified outpost behavior MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Meta-header routing and forward_auth removed from the described design — both disproved live. Documents: Host-based provider routing with per-guarded-host proxy providers (lockstep rule), the null-flows wedge that starves the embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next URL as a dead design, and the public-DNS record requirement per guarded host. --- .../Architecture/Network Architecture.md | 26 ++-- .../Infrastructure/Anubis.md | 33 +++-- .../Infrastructure/Authentik.md | 126 +++++++++++------- 3 files changed, 116 insertions(+), 69 deletions(-) diff --git a/serverAppConfigs-docs/Architecture/Network Architecture.md b/serverAppConfigs-docs/Architecture/Network Architecture.md index eb8da25..b85c364 100644 --- a/serverAppConfigs-docs/Architecture/Network Architecture.md +++ b/serverAppConfigs-docs/Architecture/Network Architecture.md @@ -7,8 +7,12 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS, Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone else), and accepted traffic returns to Caddy's internal listener on `:8016`, -where per-vhost routing resumes. Authentik's outpost then gates the wrapped -apps with `forward_auth`. See [[Anubis]] and [[Authentik]]. +which proxies every request whole to the Authentik server with the Host +header preserved; Authentik's embedded outpost routes on that Host against +each proxy provider's `external_host`, serves the login flow in place for +sessionless clients, and forwards authenticated traffic back to Caddy's +post-auth listener `:8017`, where per-vhost routing reaches the app. See +[[Anubis]] and [[Authentik]]. ```mermaid graph TB @@ -19,12 +23,13 @@ graph TB CADDY -->|unwrapped vhosts route direct| DIRECT{direct routes} CADDY -->|guarded vhosts| ANUBIS[Anubis verify :8923] ANUBIS -->|deny AI bots| EXT - ANUBIS -->|challenge solved or bypassed| INT[Caddy internal listener :8016] - INT -->|forward_auth outpost| AUTH[Authentik server :9000 loopback] - INT --> HOME[Homepage file_server] - INT --> SEARCH[SearXNG via Gluetun netns :8081] - INT --> HERMES[Hermes :8090] - INT --> PROM[Prometheus :9090] + ANUBIS -->|challenge solved or transport-exempt| INT[Caddy internal listener :8016] + INT -->|whole request, Host preserved| AUTH[Authentik embedded outpost :9000] + AUTH -->|session valid, provider internal_host| POST[Caddy post-auth listener :8017] + POST --> HOME[Homepage file_server] + POST --> SEARCH[SearXNG via Gluetun netns :8081] + POST --> HERMES[Hermes :8090] + POST --> PROM[Prometheus :9090] DIRECT -->|3000 + ssh 2222| GITEA[Gitea] DIRECT -->|3005| OWEBUI[OpenWebUI] DIRECT -->|3002| GRAFANA[Grafana :3000] @@ -43,7 +48,8 @@ graph TB ARR[*arr stack + qBittorrent] end - INT -->|romm, files guarded paths| ROMM + POST -->|romm| ROMM + POST -->|files| COPY DIRECT -->|plex plain-HTTP passthrough| PLEX DIRECT --> CAL DIRECT --> JEL @@ -77,7 +83,7 @@ SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetu ## Access Policy - The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state. -- Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (`forward_auth`) gates the wrapped apps. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]]. +- Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (whole-request proxy, routed by `Host` → provider `external_host`) gates the wrapped apps — one proxy provider per guarded host is mandatory, and Anubis exempts the `/outpost.goauthentik.io/*` transport paths so login can happen before a challenge cookie exists. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]]. - LAN subnet: `192.168.0.0/24`. ## See Also diff --git a/serverAppConfigs-docs/Infrastructure/Anubis.md b/serverAppConfigs-docs/Infrastructure/Anubis.md index ad13f03..1ebd74f 100644 --- a/serverAppConfigs-docs/Infrastructure/Anubis.md +++ b/serverAppConfigs-docs/Infrastructure/Anubis.md @@ -4,9 +4,8 @@ Anubis is the bot-challenge proxy that fronts the guarded public vhosts on Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are denied outright. It sits in a two-tier chain: client → Caddy TLS listener → Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app. -Host headers survive both hops, so per-vhost routing (and Authentik -`forward_auth` for the wrapped apps) resumes unchanged behind it. See -[[Caddy]] and [[Authentik]]. +Host headers survive both hops, so the Authentik embedded outpost can route +on the original `Host` behind it. See [[Caddy]] and [[Authentik]]. ```mermaid graph LR @@ -14,7 +13,7 @@ graph LR T --> A[Anubis verify :8923] A -->|accepted| I[Caddy internal listener :8016] A -->|challenge or deny| C - I --> P[per-vhost routing and forward_auth] + I --> P[whole-request proxy to Authentik outpost] ``` ## Facts (live inventory) @@ -31,10 +30,14 @@ graph LR Policy content: an `ALLOW` rule for Anubis's own endpoints (the `/.within.website/x/cmd/anubis/*` API and static namespace — the challenge JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging -those endpoints makes the challenge challenge itself), then DENY for -AI-scraper user-agent regexes (GPTBot, Claude-*, Perplexity*, CCBot, -Bytespider, Amazonbot, meta-externalagent, and friends), then CHALLENGE for -everything else (`path_regex: "."` catch-all). DNSBL off. Valid action verbs +those endpoints makes the challenge challenge itself), an `ALLOW` rule for +the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start, +callback and the outpost's static assets must be reachable before any +session or challenge dance completes — authentik's own session gate protects +them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*, +Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends), +then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off. +Valid action verbs at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them (first deploy of that rule crash-looped the unit on exactly that typo; verified against the image binary's string table). The policy schema at this @@ -49,7 +52,7 @@ These never route through Anubis (no policy bypass rule needed): plex ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — `/info/refs` and `git-*-pack` paths pass unchallenged). -## Gotchas (all verified live 2026-08-28) +## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30) 1. **Image ref casing.** Podman's reference parser rejects mixed-case repository refs before any network I/O (`repository name must be @@ -87,6 +90,16 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — login flow. Verify with a JS-capable client end-to-end — mint via `make-challenge`, solve, submit with browser-exact query params — before declaring a challenge chain healthy. +7. **Single-file bind mount + git pull = stale policy.** The policy is + mounted file-at-a-time (`/home/dank/serverAppConfigs/dankclaw/anubis/ + policy.json:/policy/policy.json:ro`); bind mounts pin the INODE. A + `git pull`/checkout that replaces the file leaves the running container + reading the old inode indefinitely (observed live: 22 h of uptime serving + a policy already superseded on disk, with no reload log line because + there is nothing watching). A policy change only takes effect when the + anubis container is (re)started — `./scripts/deploy.sh dankclaw + --service anubis` — and the deployer must therefore always restart it + rather than trust the file contents on disk. ## Verify live @@ -94,4 +107,6 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — ssh dankclaw 'systemctl --user is-active anubis.service' curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://franksdank.cloud/ # denied page curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://franksdank.cloud/ # challenge 200 +curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt + ``` diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index 8f95033..3028740 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -1,12 +1,12 @@ # Authentik — the homelab's SSO / OIDC provider Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the -homelab. It fronts the Caddy vhosts that do not carry their own -application-level authentication. Apps that have their own login (Grafana, -Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in -forward_auth — they handle their own session. +homelab. It protects the Caddy vhosts that do not carry their own +application-level authentication with its embedded proxy outpost. Apps that +have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack, +Pi-hole) are not wrapped — they handle their own session. -- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`) +- **Version:** `2026.8.0` (pinned in `dankclaw/authentik/quadlet/*.container`) - **Host:** dankclaw (`192.168.0.143`) - **Name:** `auth.franksdank.cloud` - **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`, @@ -16,9 +16,11 @@ forward_auth — they handle their own session. The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in `.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI -for all configuration, a native Caddy `forward_auth` outpost, and a footprint -that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10, -so the stack is 3 services, not 4. +for all configuration, an outpost that terminates auth for reverse-proxied +apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis +was removed in 2025.10, so the stack is 3 services, not 4. (The integration +mechanism changed in 2026.x — see Architecture; the research file predates +the pin.) ## Architecture @@ -26,25 +28,31 @@ so the stack is 3 services, not 4. graph LR U[Browser] -->|1. GET /search?q=x| A[Anubis :8923
bot challenge] A -->|2. verified, dials internal listener| C[Caddy :8016
auth decision] - C -->|3. whole request + meta headers| S[authentik-server :9000
proxy provider 1] + C -->|3. whole request, Host preserved| S[authentik-server :9000
embedded outpost] S -->|4. no session: login flow served in place| U S -->|5. session valid: proxies on to internal_host| P[Caddy :8017
post-auth routing] P -->|6. app response back up the chain| U S --> PG[(postgres)] ``` -In authentik 2026.8 the proxy provider is an OAuth2 provider and the old -`/outpost.goauthentik.io/auth/caddy` forward-auth endpoint no longer exists — -every request that reaches it returns 404 (verified live, 2026-08-29). The -chain therefore does not forward-auth *to* a path: guarded vhosts proxy the -**whole request through authentik** with `X-authentik-meta-outpost`, -`X-authentik-meta-provider` and `X-authentik-meta-app` headers. The provider's -`authentication_flow` (`default-authentication-flow`) decides: no session → -the login flow is served in place; session valid → the policy passes and -authentik proxies the request on to the provider's `internal_host` -(`http://caddy:8017`, Caddy's post-auth internal listener), which routes it -straight to the app. The chain terminates — nothing dials back into -Authentik from `:8017`. +Two mechanisms were disproved live (2026-08-29/30) and one is the actual +contract: + +- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not + exist in the 2026.8 URL tree — every unauthenticated subrequest got + authentik's own 404 page. +- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing + in this build: a request carrying Caddy's full meta-header set still 404s, + while the same request with only a matching `Host` answers. The embedded + outpost routes **solely on the original `Host`**, matched against each + attached proxy provider's `external_host`. +- Consequence: **every guarded host needs its own proxy provider** (exact + `external_host`), and `:8016` is a single uniform `reverse_proxy` to + `authentik-server:9000` that preserves `Host`. Unauthenticated → login flow + served in place; authenticated → forwarded to the provider's + `internal_host` (`http://caddy:8017`), which routes straight to the app. + The chain terminates there — nothing dials back into Authentik from + `:8017`. ## The three compose services @@ -65,39 +73,51 @@ Authentik from `:8017`. ## Caddy integration -Guarded vhosts proxy the **whole request through Authentik** on Caddy's -internal listener — there is no `forward_auth` anywhere: in 2026.8 the auth -endpoint the old config pointed at (`/outpost.goauthentik.io/auth/caddy`) -does not exist in the URL tree, so every unauthenticated subrequest got a -404 and browsers were served authentik's own Not-Found page (verified live, -2026-08-29). Pattern (one example; the other guarded vhosts are identical): +Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which +proxies **every request whole** to the Authentik server with `Host` +preserved — there is no `forward_auth` and no per-vhost block: ```caddyfile http://:8016 { - @search host search.franksdank.cloud - handle @search { - # Unauthenticated: the login flow is served in place. - # Authenticated: Authentik proxies on to internal_host (:8017). - reverse_proxy authentik-server:9000 { - header_up X-authentik-meta-outpost 7e39655b-58d3-4fe6-a80a-513f31051217 - header_up X-authentik-meta-provider 1 - header_up X-authentik-meta-app name=caddy - } + reverse_proxy authentik-server:9000 { + header_up X-Forwarded-Proto https } } ``` -The meta headers tell authentik which outpost, provider and application are -answering (the values come from the DB: outpost pk = the UUID embedded in -`AUTHENTIK_OUTPOST_TOKEN`, provider pk 1 = `embedded-outpost`, app slug -`caddy`). The provider's `internal_host` is `http://caddy:8017`, so -authenticated traffic is handed back to Caddy's post-auth listener, which -routes it straight to the app — the chain terminates there. +The embedded outpost inside the server process decides by `Host`: +no session → the provider's `authentication_flow` is served in place; +session valid → the request is forwarded to the provider's `internal_host` +(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps. + +**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017` +routing table, and the Authentik proxy-provider list must be the same set of +hosts. A guarded host with no provider 404s at login time; a provider with no +DNS record breaks off-LAN clients. + +Live provider inventory (provisioned via ORM, 2026-08-30), all attached to +outpost `7e39655b-…` ("authentik Embedded Outpost"), all with +`internal_host = http://caddy:8017` and all three flows set: + +| Provider pk | name | external_host | app slug | +|---|---|---|---| +| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy | +| 2 | search-proxy | `https://search.franksdank.cloud` | search | +| 3 | hermes-proxy | `https://hermes.franksdank.cloud` | hermes | +| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus | +| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm | +| 6 | files-proxy | `https://files.franksdank.cloud` | files | + +Provisioning pitfall (verified live): a ProxyProvider created with only +`authentication_flow` set — `authorization_flow`/`invalidation_flow` null — +makes the embedded outpost thread fail its own config fetch: the Rust side +logs `Failed to fetch outpost from API, retrying in 3 seconds` forever and +the entire `/outpost.goauthentik.io` space 404s for every host. Backfill all +three flows (pk1's values: `default-authentication-flow`, +`default-provider-authorization-implicit-consent`, +`default-provider-invalidation-flow`) and the retry loop recovers on its next +3-second cycle. -Guarded vhosts: the homepage (`franksdank.cloud`, `www.franksdank.cloud`), -SearXNG, Hermes, Prometheus, RomM, Copyparty. The `auth.franksdank.cloud` -vhost proxies to the server directly — it *is* the login, and it must never -sit behind a challenge. ## Configuration (secrets) @@ -113,9 +133,12 @@ Two secrets live in the host `.env` (git-ignored; placeholders in health-gated), reloads Caddy. 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema. -3. In the admin interface: **Directory → Applications** → add a **Proxy** - application (or an **OAuth2/OpenID Connect** provider) for each vhost you - protect. The Caddy outpost reads the application's policy. +3. For every guarded vhost, in the admin interface: **Directory → + Applications** → add an application + a **Proxy** provider whose + `external_host` is exactly the browser-facing origin + (`https://`), `internal_host = http://caddy:8017`, and **all three** + flows set (see the provisioning pitfall above). Attach the provider to the + embedded outpost. 4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID Connect** provider, create the app with the correct redirect URI, and paste the `client_id` / `client_secret` into the app's SSO settings. @@ -138,10 +161,13 @@ Two secrets live in the host `.env` (git-ignored; placeholders in | Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` | | "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy | | Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir | +| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie | +| `/outpost.goauthentik.io/*` 404s on every host | A provider row with null `authorization_flow`/`invalidation_flow` wedges the outpost config fetch | Backfill the flows; watch `podman logs authentik-server` for `Failed to fetch outpost from API` | +| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer | ## Cross-references -- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives +- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners - [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud` - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme - Research: `.docaudit/research/auth.md` — the full provider comparison