serverAppConfigs/serverAppConfigs-docs/Infrastructure/Anubis.md

Anubis — bot challenge proxy

Anubis is the bot-challenge proxy that fronts the guarded public vhosts on Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are denied outright. It sits in a two-tier chain: client → Caddy TLS listener → Anubis (:8923, internal-only) → Caddy internal listener (:8016) → app. Host headers survive both hops, so the Authentik embedded outpost can route on the original Host behind it. See Caddy and Authentik.

graph LR
    C[Client] --> T[Caddy TLS listener]
    T --> A[Anubis verify :8923]
    A -->|accepted| I[Caddy internal listener :8016]
    A -->|challenge or deny| C
    I --> P[whole-request proxy to Authentik outpost]

Facts (live inventory)

Item	Value
Runtime	Quadlet unit anubis.service (dankclaw/anubis/quadlet/anubis.container) — boot-persistent, self-healing
Image	ghcr.io/techarohq/anubis:v1.13.0, run under a localhost/ alias tag (see gotcha 1)
Ports	:8923 internal (no published port); metrics on :9090 reachable only inside dankclaw_default
Config surface	flags via flagenv — TARGET=http://caddy:8016 sets --target, POLICY_FNAME=/policy/policy.json sets --policy-fname
Policy	dankclaw/anubis/policy.json mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart)
Store	memory (no signing keys needed); a restart invalidates issued challenge JWTs — users simply re-challenge once

Policy content: an ALLOW rule for Anubis's own endpoints (the /.within.website/x/cmd/anubis/* API and static namespace — the challenge JS calls make-challenge/pass-challenge under that prefix, and challenging those endpoints makes the challenge challenge itself), an ALLOW rule for the Authentik transport prefix ^/outpost\.goauthentik\.io/ (login, start, callback and the outpost's static assets must be reachable before any session or challenge dance completes — authentik's own session gate protects them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-, Perplexity, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends), then CHALLENGE for everything else (path_regex: "." catch-all). DNSBL off. Valid action verbs at this tag are ALLOW, CHALLENGE, DENY — ACCEPT is not one of them (first deploy of that rule crash-looped the unit on exactly that typo; verified against the image binary's string table). The policy schema at this tag is JSON-only — no (data)/ imports and no CEL; first match wins. Upgrading the pin = stop the unit, pull + re-tag the localhost/ alias, restart; re-verify the policy schema against the new tag first.

Exempt by construction

These never route through Anubis (no policy bypass rule needed): plex (plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — /info/refs and git-*-pack paths pass unchallenged).

Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30)

1. Image ref casing. Podman's reference parser rejects mixed-case repository refs before any network I/O (repository name must be lowercase). All refs use the lowercase ghcr.io/techarohq/anubis form — ghcr resolves case-insensitively at pull time.
2. Quadlet + registry-prefixed refs. On this host's podman 5.7 quadlet path, a container created from a registry-prefixed image ref fails with image not known even when the image is present in storage; the identical argv works from a shell or a transient unit. The unit therefore runs the image under a localhost/techarohq/anubis:v1.13.0 alias tag — deploy pulls the canonical ref, then podman tags the alias (idempotent).
3. Health probe exec form. The image is a distroless ko-build (no shell; entrypoint /ko-app/anubis). This quadlet generator joins HealthCmd into one \x20-escaped argv element and podman then runs it shell-form — exec of a missing shell exits 1 and the container sits unhealthy. A JSON exec-form array (HealthCmd=["/ko-app/anubis","--healthcheck"]) passes through the generator verbatim and executes the binary directly.
4. Rollback nuance. systemctl --user stop anubis.service makes every guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not forward_auth — Caddy's public listeners proxy to :8923). A true rollback is a Caddyfile revert + redeploy of the caddy service.
5. Deploy handler pull guard. deploy_anubis guards the pull with podman image exists; a cold anonymous ghcr token request can 403 on the first fetch — retry once, the auth cache makes the second attempt local.
6. JS-capable clients are the canary. Non-JS clients (curl, bots, scrapers) never solve challenges — they just receive the challenge page, so every curl probe looks green while the verification step is broken. The first JS-capable client to traverse a freshly deployed chain is the first real test of it: browsers POST-solve via make-challenge and GET-navigate pass-challenge?response=<sha256(challenge+nonce)>&nonce=N &elapsedTime=T&redir=<path>; if any own-endpoint path is challenged instead of allowed, the browser's fetch throws and it renders its own error page (iOS/macOS browsers phrase it as "the request failed and the interceptors did not return an alternative response") instead of the login flow. Verify with a JS-capable client end-to-end — mint via make-challenge, solve, submit with browser-exact query params — before declaring a challenge chain healthy.
7. Single-file bind mount + git pull = stale policy. The policy is mounted file-at-a-time (/home/dank/serverAppConfigs/dankclaw/anubis/ policy.json:/policy/policy.json:ro); bind mounts pin the INODE. A git pull/checkout that replaces the file leaves the running container reading the old inode indefinitely (observed live: 22 h of uptime serving a policy already superseded on disk, with no reload log line because there is nothing watching). A policy change only takes effect when the anubis container is (re)started — ./scripts/deploy.sh dankclaw --service anubis — and the deployer must therefore always restart it rather than trust the file contents on disk.

Verify live

ssh dankclaw 'systemctl --user is-active anubis.service'
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://franksdank.cloud/   # denied page
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://franksdank.cloud/  # challenge 200
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt