serverAppConfigs/serverAppConfigs-docs/Infrastructure/Anubis.md

---
type: reference
description: "Anubis is the bot-challenge proxy that fronts the guarded public vhosts on"
tags:
  - infrastructure
  - anubis
timestamp: 2026-09-01
---

# Anubis — bot challenge proxy

Anubis is the bot-challenge proxy that fronts the guarded public vhosts on
Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are
denied outright. It sits in a two-tier chain: client → Caddy TLS listener →
Anubis (`:8923`, internal-only) → Caddy internal listener (`:8016`) → app.
Host headers survive both hops, so the Authentik embedded outpost can route
on the original `Host` behind it. Since v1.27.0 the challenge cookie is
domain-wide (`COOKIE_DOMAIN=franksdank.cloud`): a browser solves the
proof-of-work once on any guarded vhost and every guarded
`*.franksdank.cloud` vhost honors it until the 168h expiry or an authentik
re-login. See [[Caddy]] and [[Authentik]].

```mermaid
graph LR
    C[Client] --> T[Caddy TLS listener]
    T --> A[Anubis verify :8923]
    A -->|accepted| I[Caddy internal listener :8016]
    A -->|challenge or deny| C
    I --> P[whole-request proxy to Authentik outpost]
```

## Facts (live inventory)

| Item | Value |
|------|-------|
| Runtime | Quadlet unit `anubis.service` (`dankclaw/anubis/quadlet/anubis.container`) — boot-persistent, self-healing |
| Image | `ghcr.io/techarohq/anubis:v1.27.0`, run under a `localhost/` alias tag (see gotcha 1) |
| Ports | `:8923` internal (no published port); metrics on `:9090` reachable only inside `dankclaw_default` |
| Config surface | flags via flagenv — `TARGET=http://caddy:8016` sets `--target`, `POLICY_FNAME=/policy/policy.json` sets `--policy-fname`, `COOKIE_DOMAIN=franksdank.cloud` sets `--cookie-domain` |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart) |
| Store | memory JWTs signed by a **persistent** ED25519 key (`ED25519_PRIVATE_KEY_HEX`, host file `/home/dank/Containers/anubis/env`, mode 0600, gitea-style `${VAR}` expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot |

Policy content: an `ALLOW` rule for Anubis's own endpoints (the
`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
those endpoints makes the challenge challenge itself), an `ALLOW` rule for
the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
callback and the outpost's static assets must be reachable before any
session or challenge dance completes — authentik's own session gate protects
them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
Valid action verbs
at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
(first deploy of that rule crash-looped the unit on exactly that typo;
verified against the image binary's string table). The policy schema at this
tag is JSON-only — no `(data)/` imports and no CEL; first match wins (JSON
still loads at v1.27.0 — scratch-verified 2026-09-02 before the pin bump).
Upgrading the pin = pull + re-tag the `localhost/` alias, restart; verify
the flag surface with `podman run --rm ... --help` and the policy against
the new tag first (done for v1.27.0).

## Exempt by construction

These never route through Anubis (no policy bypass rule needed): plex
(plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay
ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
`/info/refs` and `git-*-pack` paths pass unchallenged).

## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)

1. **Image ref casing.** Podman's reference parser rejects mixed-case
   repository refs before any network I/O (`repository name must be
   lowercase`). All refs use the lowercase `ghcr.io/techarohq/anubis` form —
   ghcr resolves case-insensitively at pull time.
2. **Quadlet + registry-prefixed refs.** On this host's podman 5.7 quadlet
   path, a container created from a registry-prefixed image ref fails with
   `image not known` even when the image is present in storage; the identical
   argv works from a shell or a transient unit. The unit therefore runs the
   image under a `localhost/techarohq/anubis:v1.13.0` alias tag — deploy
   pulls the canonical ref, then `podman tag`s the alias (idempotent).
3. **Health probe exec form.** The image is a distroless ko-build (no shell;
   entrypoint `/ko-app/anubis`). This quadlet generator joins `HealthCmd`
   into one `\x20`-escaped argv element and podman then runs it shell-form —
   exec of a missing shell exits 1 and the container sits unhealthy. A JSON
   exec-form array (`HealthCmd=["/ko-app/anubis","--healthcheck"]`) passes
   through the generator verbatim and executes the binary directly.
4. **Rollback nuance.** `systemctl --user stop anubis.service` makes every
   guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not
   forward_auth — Caddy's public listeners proxy to `:8923`). A true rollback
   is a Caddyfile revert + redeploy of the caddy service.
5. **Deploy handler pull guard.** `deploy_anubis` guards the pull with
   `podman image exists`; a cold anonymous ghcr token request can 403 on the
   first fetch — retry once, the auth cache makes the second attempt local.
6. **JS-capable clients are the canary.** Non-JS clients (curl, bots,
   scrapers) never solve challenges — they just receive the challenge page,
   so every curl probe looks green while the verification step is broken.
   The first JS-capable client to traverse a freshly deployed chain is the
   first real test of it: browsers POST-solve via `make-challenge` and
   GET-navigate `pass-challenge?response=<sha256(challenge+nonce)>&nonce=N
   &elapsedTime=T&redir=<path>`; if any own-endpoint path is challenged
   instead of allowed, the browser's fetch throws and it renders its own
   error page (iOS/macOS browsers phrase it as "the request failed and the
   interceptors did not return an alternative response") instead of the
   login flow. Verify with a JS-capable client end-to-end — mint via
   `make-challenge`, solve, submit with browser-exact query params — before
   declaring a challenge chain healthy.
7. **Single-file bind mount + git pull = stale policy.** The policy is
   mounted file-at-a-time (`/home/dank/serverAppConfigs/dankclaw/anubis/
   policy.json:/policy/policy.json:ro`); bind mounts pin the INODE. A
   `git pull`/checkout that replaces the file leaves the running container
   reading the old inode indefinitely (observed live: 22 h of uptime serving
   a policy already superseded on disk, with no reload log line because
   there is nothing watching). A policy change only takes effect when the
   anubis container is (re)started — `./scripts/deploy.sh dankclaw
   --service anubis` — and the deployer must therefore always restart it
   rather than trust the file contents on disk.

8. **X-Real-Ip is mandatory at v1.27+ (HTTP 500 without it).** Anubis now
   answers any request lacking `X-Real-Ip` with a 500 whose log line reads
   `[misconfiguration] X-Real-Ip header is not set`. The real chain always
   sets it (every guarded Caddy listener carries `header_up X-Real-Ip
   {remote_host}` — the bats seam enforces the line), so users never see
   this. But any direct probe of `:8923` (scratch container, curl, a new
   reverse-proxy hop) MUST set the header or it looks like a policy failure.
   Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.
9. **DENY renders as an error-styled page at HTTP 200.** At v1.27 the denied
   client gets the "Oh noes!" template containing `Access Denied` and
   `error code <rule-id>` — with status **200**, not 403 (the boot log maps
   rule names to error ids: `rule-error-ids`). Status-only probes cannot
   distinguish challenge from deny at this release; grep the body for
   `Access Denied` / `Making sure you` instead.

## Guard-chain layer diagnosis (a user reports a guarded page "broken")

Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
"broken / links gone / not loading" and the fault must be localized to
server-side or client-side/session. Chain topology:
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
→ Caddy :8017 (@home/@data handlers) → app/static`.

Layer-by-layer checks (run from the host via ssh):

1. **Caddy unit + guard containers**: `systemctl --user is-active
   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
2. **Anubis challenge page validity**: `curl -sk
   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
   — a healthy challenge contains `"anubis_challenge"` JSON with
   `difficulty` and the main JS.
3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
   logs are normal.
4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
   connection refused, which is EXPECTED, not a fault):
   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
   this container has NO `-H` flag — use `--header "Host: …"`.)
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).

Tool/behavior notes: the anubis container is distroless — `podman exec anubis
sh` fails (`sh: executable not found`); inspect via its logs or the served
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
podman IP) for ALL clients — rootless-podman published-port SNAT;
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
mistake that for the bug. Homepage render: `app.js` does
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
failure the catch sets `#summary` to `status unavailable: <err>` and the
grid stays EMPTY — which a user reads as "links gone / not loading." A
30-service status.json with a 9-service services.json renders fine
(stateByName lookup).

**Decision:** if every layer is healthy (challenge valid, outpost clean,
handlers 200, files render), the fault is client-side/session. STOP
thrashing the live system — repeated failed browser challenges share the
masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
symptom (challenge spinning? login loop? empty grid? "status unavailable"
text?) or the authentik login to drive the page in a real browser.

## Verify live

```
ssh dankclaw 'systemctl --user is-active anubis.service'
podman inspect anubis --format "{{.ImageName}} {{.State.Health.Status}}"
# the apex 308s to www; the guarded vhost is www — probe www, not the apex
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -A "Mozilla/5.0" https://franksdank.cloud/
# challenge: body carries the version marker + PoW page
curl -s -A "Mozilla/5.0" https://www.franksdank.cloud/ | grep -oE "cachebuster=v[0-9.]+|Making sure you.re not a bot"
# deny: HTTP 200 at v1.27+ — grep the body, not the status (gotcha 9)
curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && echo denied || echo BROKEN-challenge-served
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
```