dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: authentik/anubis/network corrected to live-verified outpost behavior

Browse Source 
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
This commit is contained in:
Frank
2026-08-29 23:55:48 -07:00
parent 2a72efad44
commit ee1e038f30
3 changed files with 116 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Architecture/Network Architecture.md
+16 -10
View File
@@ -7,8 +7,12 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS, As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
else), and accepted traffic returns to Caddy's internal listener on `:8016`, else), and accepted traffic returns to Caddy's internal listener on `:8016`,
where per-vhost routing resumes. Authentik's outpost then gates the wrapped which proxies every request whole to the Authentik server with the Host
apps with `forward_auth`. See [[Anubis]] and [[Authentik]]. header preserved; Authentik's embedded outpost routes on that Host against
each proxy provider's `external_host`, serves the login flow in place for
sessionless clients, and forwards authenticated traffic back to Caddy's
post-auth listener `:8017`, where per-vhost routing reaches the app. See
[[Anubis]] and [[Authentik]].
```mermaid ```mermaid
graph TB graph TB
@@ -19,12 +23,13 @@ graph TB
CADDY -->|unwrapped vhosts route direct| DIRECT{direct routes} CADDY -->|unwrapped vhosts route direct| DIRECT{direct routes}
CADDY -->|guarded vhosts| ANUBIS[Anubis verify :8923] CADDY -->|guarded vhosts| ANUBIS[Anubis verify :8923]
ANUBIS -->|deny AI bots| EXT ANUBIS -->|deny AI bots| EXT
ANUBIS -->|challenge solved or bypassed| INT[Caddy internal listener :8016] ANUBIS -->|challenge solved or transport-exempt| INT[Caddy internal listener :8016]
INT -->|forward_auth outpost| AUTH[Authentik server :9000 loopback] INT -->|whole request, Host preserved| AUTH[Authentik embedded outpost :9000]
INT --> HOME[Homepage file_server] AUTH -->|session valid, provider internal_host| POST[Caddy post-auth listener :8017]
INT --> SEARCH[SearXNG via Gluetun netns :8081] POST --> HOME[Homepage file_server]
INT --> HERMES[Hermes :8090] POST --> SEARCH[SearXNG via Gluetun netns :8081]
INT --> PROM[Prometheus :9090] POST --> HERMES[Hermes :8090]
POST --> PROM[Prometheus :9090]
DIRECT -->|3000 + ssh 2222| GITEA[Gitea] DIRECT -->|3000 + ssh 2222| GITEA[Gitea]
DIRECT -->|3005| OWEBUI[OpenWebUI] DIRECT -->|3005| OWEBUI[OpenWebUI]
DIRECT -->|3002| GRAFANA[Grafana :3000] DIRECT -->|3002| GRAFANA[Grafana :3000]
@@ -43,7 +48,8 @@ graph TB
ARR[*arr stack + qBittorrent] ARR[*arr stack + qBittorrent]
end end
INT -->|romm, files guarded paths| ROMM POST -->|romm| ROMM
POST -->|files| COPY
DIRECT -->|plex plain-HTTP passthrough| PLEX DIRECT -->|plex plain-HTTP passthrough| PLEX
DIRECT --> CAL DIRECT --> CAL
DIRECT --> JEL DIRECT --> JEL
@@ -77,7 +83,7 @@ SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetu
## Access Policy ## Access Policy
- The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state. - The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state.
- Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (`forward_auth`) gates the wrapped apps. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]]. - Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (whole-request proxy, routed by `Host` → provider `external_host`) gates the wrapped apps — one proxy provider per guarded host is mandatory, and Anubis exempts the `/outpost.goauthentik.io/*` transport paths so login can happen before a challenge cookie exists. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]].
- LAN subnet: `192.168.0.0/24`. - LAN subnet: `192.168.0.0/24`.
## See Also ## See Also
serverAppConfigs-docs/Infrastructure/Anubis.md
+24 -9
View File
@@ -4,9 +4,8 @@ Anubis is the bot-challenge proxy that fronts the guarded public vhosts on
Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are
denied outright. It sits in a two-tier chain: client → Caddy TLS listener → denied outright. It sits in a two-tier chain: client → Caddy TLS listener →
Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app. Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app.
Host headers survive both hops, so per-vhost routing (and Authentik Host headers survive both hops, so the Authentik embedded outpost can route
`forward_auth` for the wrapped apps) resumes unchanged behind it. See on the original `Host` behind it. See [[Caddy]] and [[Authentik]].
[[Caddy]] and [[Authentik]].
```mermaid ```mermaid
graph LR graph LR
@@ -14,7 +13,7 @@ graph LR
T --> A[Anubis verify :8923] T --> A[Anubis verify :8923]
A -->|accepted| I[Caddy internal listener :8016] A -->|accepted| I[Caddy internal listener :8016]
A -->|challenge or deny| C A -->|challenge or deny| C
I --> P[per-vhost routing and forward_auth] I --> P[whole-request proxy to Authentik outpost]
``` ```
## Facts (live inventory) ## Facts (live inventory)
@@ -31,10 +30,14 @@ graph LR
Policy content: an `ALLOW` rule for Anubis's own endpoints (the Policy content: an `ALLOW` rule for Anubis's own endpoints (the
`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge `/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
those endpoints makes the challenge challenge itself), then DENY for those endpoints makes the challenge challenge itself), an `ALLOW` rule for
AI-scraper user-agent regexes (GPTBot, Claude-*, Perplexity*, CCBot, the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
Bytespider, Amazonbot, meta-externalagent, and friends), then CHALLENGE for callback and the outpost's static assets must be reachable before any
everything else (`path_regex: "."` catch-all). DNSBL off. Valid action verbs session or challenge dance completes — authentik's own session gate protects
them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
Valid action verbs
at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
(first deploy of that rule crash-looped the unit on exactly that typo; (first deploy of that rule crash-looped the unit on exactly that typo;
verified against the image binary's string table). The policy schema at this verified against the image binary's string table). The policy schema at this
@@ -49,7 +52,7 @@ These never route through Anubis (no policy bypass rule needed): plex
ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
`/info/refs` and `git-*-pack` paths pass unchallenged). `/info/refs` and `git-*-pack` paths pass unchallenged).
## Gotchas (all verified live 2026-08-28) ## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30)
1. **Image ref casing.** Podman's reference parser rejects mixed-case 1. **Image ref casing.** Podman's reference parser rejects mixed-case
repository refs before any network I/O (`repository name must be repository refs before any network I/O (`repository name must be
@@ -87,6 +90,16 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
login flow. Verify with a JS-capable client end-to-end — mint via login flow. Verify with a JS-capable client end-to-end — mint via
`make-challenge`, solve, submit with browser-exact query params — before `make-challenge`, solve, submit with browser-exact query params — before
declaring a challenge chain healthy. declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
mounted file-at-a-time (`/home/dank/serverAppConfigs/dankclaw/anubis/
policy.json:/policy/policy.json:ro`); bind mounts pin the INODE. A
`git pull`/checkout that replaces the file leaves the running container
reading the old inode indefinitely (observed live: 22 h of uptime serving
a policy already superseded on disk, with no reload log line because
there is nothing watching). A policy change only takes effect when the
anubis container is (re)started — `./scripts/deploy.sh dankclaw
--service anubis` — and the deployer must therefore always restart it
rather than trust the file contents on disk.
## Verify live ## Verify live
@@ -94,4 +107,6 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
ssh dankclaw 'systemctl --user is-active anubis.service' ssh dankclaw 'systemctl --user is-active anubis.service'
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://franksdank.cloud/ # denied page curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://franksdank.cloud/ # denied page
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://franksdank.cloud/ # challenge 200 curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://franksdank.cloud/ # challenge 200
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
``` ```
serverAppConfigs-docs/Infrastructure/Authentik.md
+76 -50
View File
@@ -1,12 +1,12 @@
# Authentik — the homelab's SSO / OIDC provider # Authentik — the homelab's SSO / OIDC provider
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It fronts the Caddy vhosts that do not carry their own homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication. Apps that have their own login (Grafana, application-level authentication with its embedded proxy outpost. Apps that
Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
forward_auth — they handle their own session. Pi-hole) are not wrapped — they handle their own session.
- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`) - **Version:** `2026.8.0` (pinned in `dankclaw/authentik/quadlet/*.container`)
- **Host:** dankclaw (`192.168.0.143`) - **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud` - **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`, - **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
@@ -16,9 +16,11 @@ forward_auth — they handle their own session.
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI `.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
for all configuration, a native Caddy `forward_auth` outpost, and a footprint for all configuration, an outpost that terminates auth for reverse-proxied
that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10, apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
so the stack is 3 services, not 4. was removed in 2025.10, so the stack is 3 services, not 4. (The integration
mechanism changed in 2026.x — see Architecture; the research file predates
the pin.)
## Architecture ## Architecture
@@ -26,25 +28,31 @@ so the stack is 3 services, not 4.
graph LR graph LR
U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge] U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision] A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
C -->|3. whole request + meta headers| S[authentik-server :9000<br/>proxy provider 1] C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
S -->|4. no session: login flow served in place| U S -->|4. no session: login flow served in place| U
S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing] S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
P -->|6. app response back up the chain| U P -->|6. app response back up the chain| U
S --> PG[(postgres)] S --> PG[(postgres)]
``` ```
In authentik 2026.8 the proxy provider is an OAuth2 provider and the old Two mechanisms were disproved live (2026-08-29/30) and one is the actual
`/outpost.goauthentik.io/auth/caddy` forward-auth endpoint no longer exists — contract:
every request that reaches it returns 404 (verified live, 2026-08-29). The
chain therefore does not forward-auth *to* a path: guarded vhosts proxy the - The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
**whole request through authentik** with `X-authentik-meta-outpost`, exist in the 2026.8 URL tree — every unauthenticated subrequest got
`X-authentik-meta-provider` and `X-authentik-meta-app` headers. The provider's authentik's own 404 page.
`authentication_flow` (`default-authentication-flow`) decides: no session → - `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
the login flow is served in place; session valid → the policy passes and in this build: a request carrying Caddy's full meta-header set still 404s,
authentik proxies the request on to the provider's `internal_host` while the same request with only a matching `Host` answers. The embedded
(`http://caddy:8017`, Caddy's post-auth internal listener), which routes it outpost routes **solely on the original `Host`**, matched against each
straight to the app. The chain terminates — nothing dials back into attached proxy provider's `external_host`.
Authentik from `:8017`. - Consequence: **every guarded host needs its own proxy provider** (exact
`external_host`), and `:8016` is a single uniform `reverse_proxy` to
`authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
served in place; authenticated → forwarded to the provider's
`internal_host` (`http://caddy:8017`), which routes straight to the app.
The chain terminates there — nothing dials back into Authentik from
`:8017`.
## The three compose services ## The three compose services
@@ -65,39 +73,51 @@ Authentik from `:8017`.
## Caddy integration ## Caddy integration
Guarded vhosts proxy the **whole request through Authentik** on Caddy's Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
internal listener — there is no `forward_auth` anywhere: in 2026.8 the auth proxies **every request whole** to the Authentik server with `Host`
endpoint the old config pointed at (`/outpost.goauthentik.io/auth/caddy`) preserved — there is no `forward_auth` and no per-vhost block:
does not exist in the URL tree, so every unauthenticated subrequest got a
404 and browsers were served authentik's own Not-Found page (verified live,
2026-08-29). Pattern (one example; the other guarded vhosts are identical):
```caddyfile ```caddyfile
http://:8016 { http://:8016 {
@search host search.franksdank.cloud reverse_proxy authentik-server:9000 {
handle @search { header_up X-Forwarded-Proto https
# Unauthenticated: the login flow is served in place.
# Authenticated: Authentik proxies on to internal_host (:8017).
reverse_proxy authentik-server:9000 {
header_up X-authentik-meta-outpost 7e39655b-58d3-4fe6-a80a-513f31051217
header_up X-authentik-meta-provider 1
header_up X-authentik-meta-app name=caddy
}
} }
} }
``` ```
The meta headers tell authentik which outpost, provider and application are The embedded outpost inside the server process decides by `Host`:
answering (the values come from the DB: outpost pk = the UUID embedded in no session → the provider's `authentication_flow` is served in place;
`AUTHENTIK_OUTPOST_TOKEN`, provider pk 1 = `embedded-outpost`, app slug session valid → the request is forwarded to the provider's `internal_host`
`caddy`). The provider's `internal_host` is `http://caddy:8017`, so (`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.
authenticated traffic is handed back to Caddy's post-auth listener, which
routes it straight to the app — the chain terminates there. **Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
Live provider inventory (provisioned via ORM, 2026-08-30), all attached to
outpost `7e39655b-…` ("authentik Embedded Outpost"), all with
`internal_host = http://caddy:8017` and all three flows set:
| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 3 | hermes-proxy | `https://hermes.franksdank.cloud` | hermes |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
Provisioning pitfall (verified live): a ProxyProvider created with only
`authentication_flow` set — `authorization_flow`/`invalidation_flow` null —
makes the embedded outpost thread fail its own config fetch: the Rust side
logs `Failed to fetch outpost from API, retrying in 3 seconds` forever and
the entire `/outpost.goauthentik.io` space 404s for every host. Backfill all
three flows (pk1's values: `default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`) and the retry loop recovers on its next
3-second cycle.
Guarded vhosts: the homepage (`franksdank.cloud`, `www.franksdank.cloud`),
SearXNG, Hermes, Prometheus, RomM, Copyparty. The `auth.franksdank.cloud`
vhost proxies to the server directly — it *is* the login, and it must never
sit behind a challenge.
## Configuration (secrets) ## Configuration (secrets)
@@ -113,9 +133,12 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
health-gated), reloads Caddy. health-gated), reloads Caddy.
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema. creates the superuser (email + password) and generates the DB schema.
3. In the admin interface: **Directory → Applications** → add a **Proxy** 3. For every guarded vhost, in the admin interface: **Directory →
application (or an **OAuth2/OpenID Connect** provider) for each vhost you Applications** → add an application + a **Proxy** provider whose
protect. The Caddy outpost reads the application's policy. `external_host` is exactly the browser-facing origin
(`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
flows set (see the provisioning pitfall above). Attach the provider to the
embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID 4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
Connect** provider, create the app with the correct redirect URI, and paste Connect** provider, create the app with the correct redirect URI, and paste
the `client_id` / `client_secret` into the app's SSO settings. the `client_id` / `client_secret` into the app's SSO settings.
@@ -138,10 +161,13 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` | | Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy | | "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir | | Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | A provider row with null `authorization_flow`/`invalidation_flow` wedges the outpost config fetch | Backfill the flows; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |
## Cross-references ## Cross-references
- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives - [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud` - [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison - Research: `.docaudit/research/auth.md` — the full provider comparison