serverAppConfigs/serverAppConfigs-docs/Infrastructure/Anubis.md

---
type: reference
description: "Anubis is the bot-challenge proxy that fronts the guarded public vhosts on"
tags:
  - infrastructure
  - anubis
timestamp: 2026-09-01
---

# Anubis — bot challenge proxy

Anubis is the bot-challenge proxy that fronts the guarded public vhosts on
Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are
denied outright. It sits in a two-tier chain: client → Caddy TLS listener →
Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app.
Host headers survive both hops, so the Authentik embedded outpost can route
on the original `Host` behind it. Since v1.27.0 the challenge cookie is
domain-wide (`COOKIE_DOMAIN=franksdank.cloud`): a browser solves the
proof-of-work once on any guarded vhost and every guarded
`*.franksdank.cloud` vhost honors it until the 168h expiry or an authentik
re-login. See [[Caddy]] and [[Authentik]].

```mermaid
graph LR
    C[Client] --> T[Caddy TLS listener]
    T --> A[Anubis verify :8923]
    A -->|accepted| I[Caddy internal listener :8016]
    A -->|challenge or deny| C
    I --> P[whole-request proxy to Authentik outpost]
```

## Facts (live inventory)

| Item | Value |
|------|-------|
| Runtime | Quadlet unit `anubis.service` (`dankclaw/anubis/quadlet/anubis.container`) — boot-persistent, self-healing |
| Image | `ghcr.io/techarohq/anubis:v1.27.0`, run under a `localhost/` alias tag (see gotcha 1) |
| Ports | `:8923` internal (no published port — `podman ps` shows no port mapping, 2026-10-18); metrics on `:9090` reachable only inside `dankclaw_default` (`dankclaw/anubis/quadlet/anubis.container:3,58-61`) |
| Config surface | flags via flagenv — `TARGET=http://caddy:8016` sets `--target`, `POLICY_FNAME=/policy/policy.json` sets `--policy-fname`, `COOKIE_DOMAIN=franksdank.cloud` sets `--cookie-domain` |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container (`anubis.container:52`); parsed at process start only — the deployer restarts the unit after every policy change (`scripts/deploy-remote.sh:1091`) and no reload is wired here *(unverified 2026-10-18: whether upstream Anubis answers any reload signal at all; upstream docs are themselves behind an Anubis challenge, so this could not be read)* |
| Store | memory JWTs signed by a **persistent** ED25519 key (`ED25519_PRIVATE_KEY_HEX`, host file `/home/dank/Containers/anubis/env`, mode 0600, gitea-style `${VAR}` expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot |

Policy content, in the file's own order — first match wins, so the DENY
really does come first (`dankclaw/anubis/policy.json`, 2026-10-18): a **DENY**
for AI-scraper user-agent regexes (GPTBot, ChatGPT-User, OAI-SearchBot,
Claude-\*, anthropic-ai, Perplexity\*, CCBot, Bytespider, Amazonbot,
meta-externalagent, Google-Extended, DuckAssistBot, cohere-ai, YouBot,
KimiBot, Applebot-Extended; lines 2-7), then an **ALLOW** for Anubis's own
endpoint namespace `^/\.within\.website` (lines 8-12 — the challenge JS calls
`make-challenge`/`pass-challenge` under that prefix, and challenging those
endpoints makes the challenge challenge itself), then an **ALLOW** for the
Authentik transport prefix `^/outpost\.goauthentik\.io/` (lines 13-17 — login,
start, callback and the outpost's static assets must be reachable before any
session or challenge dance completes; authentik's own session gate protects
them), then **CHALLENGE** for everything else (`path_regex: "."` catch-all,
lines 18-22). DNSBL off (`dnsbl: false`, line 23). Live on 2026-10-18: a
`Mozilla/5.0` request to `https://www.franksdank.cloud/` returns 200 carrying
`anubis_challenge` and the `cachebuster=v1.27.0` marker, while
`-A GPTBot` returns the deny body — see gotcha 9 for why that is also a 200.

Valid action verbs in this file are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT`
is not one of them (first deploy of that rule crash-looped the unit on
exactly that typo; verified against the image binary's string table). The
local policy uses only the JSON rule form. Upstream documents the policy file
as YAML with JSON accepted because it is valid YAML "as of version v1.17.0 or
later" (https://anubis.techaro.lol/docs/admin/policies/), and import
statements exist in the 1.x line (https://github.com/TecharoHQ/anubis/releases
lists "allow wildcard matches in import statements" on the road to v1.27.0) —
so the older local statement "JSON-only, no `(data)/` imports, no CEL at this
tag" is a v1.13-era claim kept here as history and is **not** a property of
v1.27.0; whether CEL expressions are accepted at this exact tag was not
verifiable *(unverified 2026-10-18)*. What is verified is that this JSON
policy loads and is enforced (container `healthy`; 136 challenges issued in the
retained log since the 01:25 boot, scraper deny confirmed by probe).
Upgrading the pin = pull + re-tag the `localhost/` alias, restart; verify
the flag surface with `podman run --rm ... --help` and the policy against
the new tag first (done for v1.27.0). Upstream is ahead of the pin: Anubis's
own documentation site served its challenge page with
`cacheBuster=v1.28.2-0.20261012194510-…` when fetched on 2026-10-18, so
v1.28.x exists.

## Exempt by construction

These never route through Anubis (no policy bypass rule needed): `auth.`
(the IdP login itself must stay ungated, `dankclaw/Caddyfile:90-99`), `gitea`
(git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
`git-*-pack` paths pass unchallenged, `dankclaw/Caddyfile:110-120`), `ai.`
OpenWebUI (own SSO + API-key auth, `dankclaw/Caddyfile:145-151`), `pihole.`
(a deliberate basicauth edge instead, `dankclaw/Caddyfile:162-171`), `hermes.`
and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
`dankclaw/Caddyfile:179-201`), `plex.` including the plain-HTTP legacy name
(`dankclaw/Caddyfile:203-238`), the media vhosts `books.`/`jelly.`
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
it is exempt by absence — and therefore unauthenticated on the LAN at
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
(`dankclaw/Caddyfile:283-423`).

## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)

1. **Image ref casing.** Podman's reference parser rejects mixed-case
   repository refs before any network I/O (`repository name must be
   lowercase`). All refs use the lowercase `ghcr.io/techarohq/anubis` form —
   ghcr resolves case-insensitively at pull time.
2. **Quadlet + registry-prefixed refs.** On this host's podman 5.7 quadlet
   path, a container created from a registry-prefixed image ref fails with
   `image not known` even when the image is present in storage; the identical
   argv works from a shell or a transient unit. The unit therefore runs the
   image under a `localhost/techarohq/anubis:v1.27.0` alias tag — deploy
   pulls the canonical ref, then `podman tag`s the alias (idempotent;
   `scripts/deploy-remote.sh:1078-1084`, and `podman inspect anubis` reports
   `localhost/techarohq/anubis:v1.27.0` on 2026-10-18). The pin used to be
   v1.13.0; that name is retired, and the unit sets `Pull=never`
   (`anubis.container:54-56`) so a boot cannot stall on ghcr rate limits.
3. **Health probe exec form.** The image is a distroless ko-build (no shell;
   entrypoint `/ko-app/anubis`). This quadlet generator joins `HealthCmd`
   into one `\x20`-escaped argv element and podman then runs it shell-form —
   exec of a missing shell exits 1 and the container sits unhealthy. A JSON
   exec-form array (`HealthCmd=["/ko-app/anubis","--healthcheck"]`) passes
   through the generator verbatim and executes the binary directly.
4. **Rollback nuance.** `systemctl --user stop anubis.service` makes every
   guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not
   forward_auth — Caddy's public listeners proxy to `:8923`). A true rollback
   is a Caddyfile revert + redeploy of the caddy service.
5. **Deploy handler pull guard.** `deploy_anubis`
   (`scripts/deploy-remote.sh:1074-1092`) guards the pull with
   `podman image exists ghcr.io/techarohq/anubis:v1.27.0`; a cold anonymous
   ghcr token request can 403 on the first fetch — retry once, the auth cache
   makes the second attempt local. The handler always ends with
   `systemctl --user restart anubis.service` + a health wait, which is what
   makes a policy change land (gotcha 7).
6. **JS-capable clients are the canary.** Non-JS clients (curl, bots,
   scrapers) never solve challenges — they just receive the challenge page,
   so every curl probe looks green while the verification step is broken.
   The first JS-capable client to traverse a freshly deployed chain is the
   first real test of it: browsers POST-solve via `make-challenge` and
   GET-navigate `pass-challenge?response=<sha256(challenge+nonce)>&nonce=N
   &elapsedTime=T&redir=<path>`; if any own-endpoint path is challenged
   instead of allowed, the browser's fetch throws and it renders its own
   error page (iOS/macOS browsers phrase it as "the request failed and the
   interceptors did not return an alternative response") instead of the
   login flow. Verify with a JS-capable client end-to-end — mint
   `make-challenge`, solve, submit with browser-exact query params — before
   declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
   mounted file-at-a-time (`/home/dank/serverAppConfigs/dankclaw/anubis/
   policy.json:/policy/policy.json:ro`); bind mounts pin the INODE. A
   `git pull`/checkout that replaces the file leaves the running container
   reading the old inode indefinitely (observed live: 22 h of uptime serving
   a policy already superseded on disk, with no reload log line because
   there is nothing watching). A policy change only takes effect when the
   anubis container is (re)started — `./scripts/deploy.sh dankclaw
   --service anubis` — and the deployer must therefore always restart it
   rather than trust the file contents on disk.

8. **X-Real-Ip is mandatory at v1.27+ (HTTP 500 without it).** Anubis now
   answers any request lacking `X-Real-Ip` with a 500 whose log line reads
   `[misconfiguration] X-Real-Ip header is not set`. The real chain always
   sets it (every guarded Caddy listener carries `header_up X-Real-Ip
   {remote_host}` — the bats seam enforces the line), so users never
   this. But any direct probe of `:8923` (scratch container, curl, a new
   reverse-proxy hop) MUST set the header or it looks like a policy failure.
   Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.
9. **DENY renders as an error-styled page at HTTP 200.** At v1.27 the denied
   client gets the "Oh noes!" template containing `Access Denied` and
   `error code <rule-id>` — with status **200**, not 403 (the boot log maps
   rule names to error ids: `rule-error-ids`). Status-only probes cannot
   distinguish challenge from deny at this release; grep the body for
   `Access Denied` / `Making sure you` instead.

## Guard-chain layer diagnosis (a user reports a guarded page "broken")

Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
"broken / links gone / not loading" and the fault must be localized to
server-side or client-side/session. Chain topology:
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
→ Caddy :8017 (@home/@data handlers) → app/static`.

Layer-by-layer checks (run from the host via ssh):

1. **Caddy unit + guard containers**: `systemctl --user is-active
   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
2. **Anubis challenge page validity**: `curl -sk
   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
   — a healthy challenge contains `"anubis_challenge"` JSON with
   `difficulty` and the main JS.
3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
   logs are normal.
4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
   connection refused, which is EXPECTED, not a fault):
   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
   this container has NO `-H` flag — use `--header "Host: …"`.)
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).

Tool/behavior notes: the anubis container is distroless — `podman exec anubis
sh` fails (`sh: executable not found`); inspect via its logs or the served
challenge page. Anubis logs the Caddy container's podman address as
`x-real-ip` for ALL clients — rootless-podman published-port SNAT; clients
are indistinguishable by IP and clearance is cookie-based. Do NOT mistake
that for the bug. The address is whatever netavark gave Caddy on
`dankclaw_default` — all 138 logged `x-real-ip` values in the container's
retained log read `10.89.1.8` (2026-10-18), and it is not a pinned address,
so never fingerprint "the Caddy hop" on a literal `10.89.1.22`-style value.
Homepage render: `app.js` does
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
failure the catch sets `#summary` to `status unavailable: <err>` and the
grid stays EMPTY — which a user reads as "links gone / not loading." A
status.json that carries states for more services than services.json lists
renders fine (stateByName lookup); today both files carry 35 entries
(`shared/homepage/services.json`, `/srv/homepage/status.json`, 2026-10-18).

**Decision:** if every layer is healthy (challenge valid, outpost clean,
handlers 200, files render), the fault is client-side/session. STOP
thrashing the live system — repeated failed browser challenges share the
masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
symptom (challenge spinning? login loop? empty grid? "status unavailable"
text?) or the authentik login to drive the page in a real browser.

## Verify live

```
ssh dankclaw 'systemctl --user is-active anubis.service'
podman inspect anubis --format "{{.ImageName}} {{.State.Health.Status}}"
# the apex 308s to www; the guarded vhost is www — probe www, not the apex
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -A "Mozilla/5.0" https://franksdank.cloud/
# challenge: body carries the version marker + PoW page
curl -s -A "Mozilla/5.0" https://www.franksdank.cloud/ | grep -oE "cachebuster=v[0-9.]+|Making sure you.re not a bot"
# deny: HTTP 200 at v1.27+ — grep the body, not the status (gotcha 9)
curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && echo denied || echo BROKEN-challenge-served
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
```

Ran on 2026-10-18 from Dankclaw with `--resolve …:443:192.168.0.143`: the apex
308s to `https://www.franksdank.cloud/`; `www.` returns 200 with
`anubis_challenge` + `cachebuster=v1.27.0`; `-A GPTBot` returns 200 whose body
contains `Access Denied`; `search./outpost.goauthentik.io/login` returns 302
(exempt, not challenged). `systemctl --user is-active anubis.service` →
`active`, `is-enabled` → `generated`, container health `healthy`.