dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: authentik/anubis/network corrected to live-verified outpost behavior

Browse Source 
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
This commit is contained in:
Frank
2026-08-29 23:55:48 -07:00
parent 2a72efad44
commit ee1e038f30
3 changed files with 116 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Architecture/Network Architecture.md
+16 -10
View File
@@ -7,8 +7,12 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
else), and accepted traffic returns to Caddy's internal listener on `:8016`,
where per-vhost routing resumes. Authentik's outpost then gates the wrapped
apps with `forward_auth`. See [[Anubis]] and [[Authentik]].
which proxies every request whole to the Authentik server with the Host
header preserved; Authentik's embedded outpost routes on that Host against
each proxy provider's `external_host`, serves the login flow in place for
sessionless clients, and forwards authenticated traffic back to Caddy's
post-auth listener `:8017`, where per-vhost routing reaches the app. See
[[Anubis]] and [[Authentik]].
```mermaid
graph TB
@@ -19,12 +23,13 @@ graph TB
CADDY -->|unwrapped vhosts route direct| DIRECT{direct routes}
CADDY -->|guarded vhosts| ANUBIS[Anubis verify :8923]
ANUBIS -->|deny AI bots| EXT
ANUBIS -->|challenge solved or bypassed| INT[Caddy internal listener :8016]
INT -->|forward_auth outpost| AUTH[Authentik server :9000 loopback]
INT --> HOME[Homepage file_server]
INT --> SEARCH[SearXNG via Gluetun netns :8081]
INT --> HERMES[Hermes :8090]
INT --> PROM[Prometheus :9090]
ANUBIS -->|challenge solved or transport-exempt| INT[Caddy internal listener :8016]
INT -->|whole request, Host preserved| AUTH[Authentik embedded outpost :9000]
AUTH -->|session valid, provider internal_host| POST[Caddy post-auth listener :8017]
POST --> HOME[Homepage file_server]
POST --> SEARCH[SearXNG via Gluetun netns :8081]
POST --> HERMES[Hermes :8090]
POST --> PROM[Prometheus :9090]
DIRECT -->|3000 + ssh 2222| GITEA[Gitea]
DIRECT -->|3005| OWEBUI[OpenWebUI]
DIRECT -->|3002| GRAFANA[Grafana :3000]
@@ -43,7 +48,8 @@ graph TB
ARR[*arr stack + qBittorrent]
end
INT -->|romm, files guarded paths| ROMM
POST -->|romm| ROMM
POST -->|files| COPY
DIRECT -->|plex plain-HTTP passthrough| PLEX
DIRECT --> CAL
DIRECT --> JEL
@@ -77,7 +83,7 @@ SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetu
## Access Policy
- The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state.
- Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (`forward_auth`) gates the wrapped apps. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]].
- Access is layered: Anubis challenges/denies bots at the front door, then Authentik's embedded outpost (whole-request proxy, routed by `Host` → provider `external_host`) gates the wrapped apps — one proxy provider per guarded host is mandatory, and Anubis exempts the `/outpost.goauthentik.io/*` transport paths so login can happen before a challenge cookie exists. Apps with their own login (Grafana, Gitea, Plex) run unwrapped. See [[Anubis]] and [[Authentik]].
- LAN subnet: `192.168.0.0/24`.
## See Also
serverAppConfigs-docs/Infrastructure/Anubis.md
+24 -9
View File
@@ -4,9 +4,8 @@ Anubis is the bot-challenge proxy that fronts the guarded public vhosts on
Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are
denied outright. It sits in a two-tier chain: client → Caddy TLS listener →
Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app.
Host headers survive both hops, so per-vhost routing (and Authentik
`forward_auth` for the wrapped apps) resumes unchanged behind it. See
[[Caddy]] and [[Authentik]].
Host headers survive both hops, so the Authentik embedded outpost can route
on the original `Host` behind it. See [[Caddy]] and [[Authentik]].
```mermaid
graph LR
@@ -14,7 +13,7 @@ graph LR
T --> A[Anubis verify :8923]
A -->|accepted| I[Caddy internal listener :8016]
A -->|challenge or deny| C
I --> P[per-vhost routing and forward_auth]
I --> P[whole-request proxy to Authentik outpost]
```
## Facts (live inventory)
@@ -31,10 +30,14 @@ graph LR
Policy content: an `ALLOW` rule for Anubis's own endpoints (the
`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
those endpoints makes the challenge challenge itself), then DENY for
AI-scraper user-agent regexes (GPTBot, Claude-*, Perplexity*, CCBot,
Bytespider, Amazonbot, meta-externalagent, and friends), then CHALLENGE for
everything else (`path_regex: "."` catch-all). DNSBL off. Valid action verbs
those endpoints makes the challenge challenge itself), an `ALLOW` rule for
the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
callback and the outpost's static assets must be reachable before any
session or challenge dance completes — authentik's own session gate protects
them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
Valid action verbs
at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
(first deploy of that rule crash-looped the unit on exactly that typo;
verified against the image binary's string table). The policy schema at this
@@ -49,7 +52,7 @@ These never route through Anubis (no policy bypass rule needed): plex
ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
`/info/refs` and `git-*-pack` paths pass unchallenged).
## Gotchas (all verified live 2026-08-28)
## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30)
1. **Image ref casing.** Podman's reference parser rejects mixed-case
repository refs before any network I/O (`repository name must be
@@ -87,6 +90,16 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
login flow. Verify with a JS-capable client end-to-end — mint via
`make-challenge`, solve, submit with browser-exact query params — before
declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
mounted file-at-a-time (`/home/dank/serverAppConfigs/dankclaw/anubis/
policy.json:/policy/policy.json:ro`); bind mounts pin the INODE. A
`git pull`/checkout that replaces the file leaves the running container
reading the old inode indefinitely (observed live: 22 h of uptime serving
a policy already superseded on disk, with no reload log line because
there is nothing watching). A policy change only takes effect when the
anubis container is (re)started — `./scripts/deploy.sh dankclaw
--service anubis` — and the deployer must therefore always restart it
rather than trust the file contents on disk.
## Verify live
@@ -94,4 +107,6 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
ssh dankclaw 'systemctl --user is-active anubis.service'
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://franksdank.cloud/ # denied page
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://franksdank.cloud/ # challenge 200
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
```
serverAppConfigs-docs/Infrastructure/Authentik.md
+76 -50
View File
@@ -1,12 +1,12 @@
# Authentik — the homelab's SSO / OIDC provider
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
homelab. It fronts the Caddy vhosts that do not carry their own
application-level authentication. Apps that have their own login (Grafana,
Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in
forward_auth — they handle their own session.
homelab. It protects the Caddy vhosts that do not carry their own
application-level authentication with its embedded proxy outpost. Apps that
have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
Pi-hole) are not wrapped — they handle their own session.
- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`)
- **Version:** `2026.8.0` (pinned in `dankclaw/authentik/quadlet/*.container`)
- **Host:** dankclaw (`192.168.0.143`)
- **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
@@ -16,9 +16,11 @@ forward_auth — they handle their own session.
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
for all configuration, a native Caddy `forward_auth` outpost, and a footprint
that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10,
so the stack is 3 services, not 4.
for all configuration, an outpost that terminates auth for reverse-proxied
apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
was removed in 2025.10, so the stack is 3 services, not 4. (The integration
mechanism changed in 2026.x — see Architecture; the research file predates
the pin.)
## Architecture
@@ -26,25 +28,31 @@ so the stack is 3 services, not 4.
graph LR
U[Browser] -->|1. GET /search?q=x| A[Anubis :8923<br/>bot challenge]
A -->|2. verified, dials internal listener| C[Caddy :8016<br/>auth decision]
C -->|3. whole request + meta headers| S[authentik-server :9000<br/>proxy provider 1]
C -->|3. whole request, Host preserved| S[authentik-server :9000<br/>embedded outpost]
S -->|4. no session: login flow served in place| U
S -->|5. session valid: proxies on to internal_host| P[Caddy :8017<br/>post-auth routing]
P -->|6. app response back up the chain| U
S --> PG[(postgres)]
```
In authentik 2026.8 the proxy provider is an OAuth2 provider and the old
`/outpost.goauthentik.io/auth/caddy` forward-auth endpoint no longer exists —
every request that reaches it returns 404 (verified live, 2026-08-29). The
chain therefore does not forward-auth *to* a path: guarded vhosts proxy the
**whole request through authentik** with `X-authentik-meta-outpost`,
`X-authentik-meta-provider` and `X-authentik-meta-app` headers. The provider's
`authentication_flow` (`default-authentication-flow`) decides: no session →
the login flow is served in place; session valid → the policy passes and
authentik proxies the request on to the provider's `internal_host`
(`http://caddy:8017`, Caddy's post-auth internal listener), which routes it
straight to the app. The chain terminates — nothing dials back into
Authentik from `:8017`.
Two mechanisms were disproved live (2026-08-29/30) and one is the actual
contract:
- The old `/outpost.goauthentik.io/auth/caddy` forward-auth endpoint does not
exist in the 2026.8 URL tree — every unauthenticated subrequest got
authentik's own 404 page.
- `X-authentik-meta-outpost/-provider/-app` headers are IGNORED for routing
in this build: a request carrying Caddy's full meta-header set still 404s,
while the same request with only a matching `Host` answers. The embedded
outpost routes **solely on the original `Host`**, matched against each
attached proxy provider's `external_host`.
- Consequence: **every guarded host needs its own proxy provider** (exact
`external_host`), and `:8016` is a single uniform `reverse_proxy` to
`authentik-server:9000` that preserves `Host`. Unauthenticated → login flow
served in place; authenticated → forwarded to the provider's
`internal_host` (`http://caddy:8017`), which routes straight to the app.
The chain terminates there — nothing dials back into Authentik from
`:8017`.
## The three compose services
@@ -65,39 +73,51 @@ Authentik from `:8017`.
## Caddy integration
Guarded vhosts proxy the **whole request through Authentik** on Caddy's
internal listener — there is no `forward_auth` anywhere: in 2026.8 the auth
endpoint the old config pointed at (`/outpost.goauthentik.io/auth/caddy`)
does not exist in the URL tree, so every unauthenticated subrequest got a
404 and browsers were served authentik's own Not-Found page (verified live,
2026-08-29). Pattern (one example; the other guarded vhosts are identical):
Guarded vhosts reach Caddy's internal listener `:8016` (post-Anubis), which
proxies **every request whole** to the Authentik server with `Host`
preserved — there is no `forward_auth` and no per-vhost block:
```caddyfile
http://:8016 {
@search host search.franksdank.cloud
handle @search {
# Unauthenticated: the login flow is served in place.
# Authenticated: Authentik proxies on to internal_host (:8017).
reverse_proxy authentik-server:9000 {
header_up X-authentik-meta-outpost 7e39655b-58d3-4fe6-a80a-513f31051217
header_up X-authentik-meta-provider 1
header_up X-authentik-meta-app name=caddy
}
reverse_proxy authentik-server:9000 {
header_up X-Forwarded-Proto https
}
}
```
The meta headers tell authentik which outpost, provider and application are
answering (the values come from the DB: outpost pk = the UUID embedded in
`AUTHENTIK_OUTPOST_TOKEN`, provider pk 1 = `embedded-outpost`, app slug
`caddy`). The provider's `internal_host` is `http://caddy:8017`, so
authenticated traffic is handed back to Caddy's post-auth listener, which
routes it straight to the app — the chain terminates there.
The embedded outpost inside the server process decides by `Host`:
no session → the provider's `authentication_flow` is served in place;
session valid → the request is forwarded to the provider's `internal_host`
(`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.
**Lockstep rule:** the Anubis guard set, the Caddy TLS edge, the `:8017`
routing table, and the Authentik proxy-provider list must be the same set of
hosts. A guarded host with no provider 404s at login time; a provider with no
DNS record breaks off-LAN clients.
Live provider inventory (provisioned via ORM, 2026-08-30), all attached to
outpost `7e39655b-…` ("authentik Embedded Outpost"), all with
`internal_host = http://caddy:8017` and all three flows set:
| Provider pk | name | external_host | app slug |
|---|---|---|---|
| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
| 2 | search-proxy | `https://search.franksdank.cloud` | search |
| 3 | hermes-proxy | `https://hermes.franksdank.cloud` | hermes |
| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
| 6 | files-proxy | `https://files.franksdank.cloud` | files |
Provisioning pitfall (verified live): a ProxyProvider created with only
`authentication_flow` set — `authorization_flow`/`invalidation_flow` null —
makes the embedded outpost thread fail its own config fetch: the Rust side
logs `Failed to fetch outpost from API, retrying in 3 seconds` forever and
the entire `/outpost.goauthentik.io` space 404s for every host. Backfill all
three flows (pk1's values: `default-authentication-flow`,
`default-provider-authorization-implicit-consent`,
`default-provider-invalidation-flow`) and the retry loop recovers on its next
3-second cycle.
Guarded vhosts: the homepage (`franksdank.cloud`, `www.franksdank.cloud`),
SearXNG, Hermes, Prometheus, RomM, Copyparty. The `auth.franksdank.cloud`
vhost proxies to the server directly — it *is* the login, and it must never
sit behind a challenge.
## Configuration (secrets)
@@ -113,9 +133,12 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
health-gated), reloads Caddy.
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
creates the superuser (email + password) and generates the DB schema.
3. In the admin interface: **Directory → Applications** → add a **Proxy**
application (or an **OAuth2/OpenID Connect** provider) for each vhost you
protect. The Caddy outpost reads the application's policy.
3. For every guarded vhost, in the admin interface: **Directory →
Applications** → add an application + a **Proxy** provider whose
`external_host` is exactly the browser-facing origin
(`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
flows set (see the provisioning pitfall above). Attach the provider to the
embedded outpost.
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
Connect** provider, create the app with the correct redirect URI, and paste
the `client_id` / `client_secret` into the app's SSO settings.
@@ -138,10 +161,13 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
| "Request has been denied: Invalid next URL" at sign-in | Flow entered with an absolute cross-host `next=` (old redirect design) | Do not hand-craft flow URLs; enter guarded hosts normally — the outpost carries state in its own cookie |
| `/outpost.goauthentik.io/*` 404s on every host | A provider row with null `authorization_flow`/`invalidation_flow` wedges the outpost config fetch | Backfill the flows; watch `podman logs authentik-server` for `Failed to fetch outpost from API` |
| Off-LAN clients get "hostname could not be found" | Guarded host missing from the public DNS zone | Every host in the lockstep set needs a public A record (Porkbun) as well as the Pi-hole LAN answer |
## Cross-references
- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives
- [Caddy](Caddy.md) — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
- Research: `.docaudit/research/auth.md` — the full provider comparison