serverAppConfigs/serverAppConfigs-docs/Infrastructure/Anubis.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Anubis is the bot-challenge proxy that fronts the guarded public vhosts on	infrastructure anubis	2026-09-01

Anubis — bot challenge proxy

Anubis is the bot-challenge proxy that fronts the guarded public vhosts on Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are denied outright. It sits in a two-tier chain: client → Caddy TLS listener → Anubis (:8923, internal-only) → Caddy internal listener (:8016) → app. Host headers survive both hops, so the Authentik embedded outpost can route on the original Host behind it. Since v1.27.0 the challenge cookie is domain-wide (COOKIE_DOMAIN=franksdank.cloud): a browser solves the proof-of-work once on any guarded vhost and every guarded *.franksdank.cloud vhost honors it until the 168h expiry or an authentik re-login. See Caddy and Authentik.

graph LR
    C[Client] --> T[Caddy TLS listener]
    T --> A[Anubis verify :8923]
    A -->|accepted| I[Caddy internal listener :8016]
    A -->|challenge or deny| C
    I --> P[whole-request proxy to Authentik outpost]

Facts (live inventory)

Item	Value
Runtime	Quadlet unit anubis.service (dankclaw/anubis/quadlet/anubis.container) — boot-persistent, self-healing
Image	ghcr.io/techarohq/anubis:v1.27.0, run under a localhost/ alias tag (see gotcha 1)
Ports	:8923 internal (no published port); metrics on :9090 reachable only inside dankclaw_default
Config surface	flags via flagenv — TARGET=http://caddy:8016 sets --target, POLICY_FNAME=/policy/policy.json sets --policy-fname, COOKIE_DOMAIN=franksdank.cloud sets --cookie-domain
Policy	dankclaw/anubis/policy.json mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart)
Store	memory JWTs signed by a persistent ED25519 key (ED25519_PRIVATE_KEY_HEX, host file /home/dank/Containers/anubis/env, mode 0600, gitea-style ${VAR} expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot

Policy content: an ALLOW rule for Anubis's own endpoints (the /.within.website/x/cmd/anubis/* API and static namespace — the challenge JS calls make-challenge/pass-challenge under that prefix, and challenging those endpoints makes the challenge challenge itself), an ALLOW rule for the Authentik transport prefix ^/outpost\.goauthentik\.io/ (login, start, callback and the outpost's static assets must be reachable before any session or challenge dance completes — authentik's own session gate protects them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-, Perplexity, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends), then CHALLENGE for everything else (path_regex: "." catch-all). DNSBL off. Valid action verbs at this tag are ALLOW, CHALLENGE, DENY — ACCEPT is not one of them (first deploy of that rule crash-looped the unit on exactly that typo; verified against the image binary's string table). The policy schema at this tag is JSON-only — no (data)/ imports and no CEL; first match wins (JSON still loads at v1.27.0 — scratch-verified 2026-09-02 before the pin bump). Upgrading the pin = pull + re-tag the localhost/ alias, restart; verify the flag surface with podman run --rm ... --help and the policy against the new tag first (done for v1.27.0).

Exempt by construction

These never route through Anubis (no policy bypass rule needed): plex (plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay ungated), and gitea (git smart-HTTP clients cannot solve JS challenges — /info/refs and git-*-pack paths pass unchallenged).

Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)

1. Image ref casing. Podman's reference parser rejects mixed-case repository refs before any network I/O (repository name must be lowercase). All refs use the lowercase ghcr.io/techarohq/anubis form — ghcr resolves case-insensitively at pull time.

2. Quadlet + registry-prefixed refs. On this host's podman 5.7 quadlet path, a container created from a registry-prefixed image ref fails with image not known even when the image is present in storage; the identical argv works from a shell or a transient unit. The unit therefore runs the image under a localhost/techarohq/anubis:v1.13.0 alias tag — deploy pulls the canonical ref, then podman tags the alias (idempotent).

3. Health probe exec form. The image is a distroless ko-build (no shell; entrypoint /ko-app/anubis). This quadlet generator joins HealthCmd into one \x20-escaped argv element and podman then runs it shell-form — exec of a missing shell exits 1 and the container sits unhealthy. A JSON exec-form array (HealthCmd=["/ko-app/anubis","--healthcheck"]) passes through the generator verbatim and executes the binary directly.

4. Rollback nuance. systemctl --user stop anubis.service makes every guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not forward_auth — Caddy's public listeners proxy to :8923). A true rollback is a Caddyfile revert + redeploy of the caddy service.

5. Deploy handler pull guard. deploy_anubis guards the pull with podman image exists; a cold anonymous ghcr token request can 403 on the first fetch — retry once, the auth cache makes the second attempt local.

6. JS-capable clients are the canary. Non-JS clients (curl, bots, scrapers) never solve challenges — they just receive the challenge page, so every curl probe looks green while the verification step is broken. The first JS-capable client to traverse a freshly deployed chain is the first real test of it: browsers POST-solve via make-challenge and GET-navigate pass-challenge?response=<sha256(challenge+nonce)>&nonce=N &elapsedTime=T&redir=<path>; if any own-endpoint path is challenged instead of allowed, the browser's fetch throws and it renders its own error page (iOS/macOS browsers phrase it as "the request failed and the interceptors did not return an alternative response") instead of the login flow. Verify with a JS-capable client end-to-end — mint via make-challenge, solve, submit with browser-exact query params — before declaring a challenge chain healthy.

7. Single-file bind mount + git pull = stale policy. The policy is mounted file-at-a-time (/home/dank/serverAppConfigs/dankclaw/anubis/ policy.json:/policy/policy.json:ro); bind mounts pin the INODE. A git pull/checkout that replaces the file leaves the running container reading the old inode indefinitely (observed live: 22 h of uptime serving a policy already superseded on disk, with no reload log line because there is nothing watching). A policy change only takes effect when the anubis container is (re)started — ./scripts/deploy.sh dankclaw --service anubis — and the deployer must therefore always restart it rather than trust the file contents on disk.

8. X-Real-Ip is mandatory at v1.27+ (HTTP 500 without it). Anubis now answers any request lacking X-Real-Ip with a 500 whose log line reads [misconfiguration] X-Real-Ip header is not set. The real chain always sets it (every guarded Caddy listener carries header_up X-Real-Ip {remote_host} — the bats seam enforces the line), so users never see this. But any direct probe of :8923 (scratch container, curl, a new reverse-proxy hop) MUST set the header or it looks like a policy failure. Scratch-verify with -H "X-Real-Ip: 192.168.0.20".

9. DENY renders as an error-styled page at HTTP 200. At v1.27 the denied client gets the "Oh noes!" template containing Access Denied and error code <rule-id> — with status 200, not 403 (the boot log maps rule names to error ids: rule-error-ids). Status-only probes cannot distinguish challenge from deny at this release; grep the body for Access Denied / Making sure you instead.

Guard-chain layer diagnosis (a user reports a guarded page "broken")

Use when a guarded page (e.g. the www.franksdank.cloud homepage) reads as "broken / links gone / not loading" and the fault must be localized to server-side or client-side/session. Chain topology: Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home/@data handlers) → app/static.

Layer-by-layer checks (run from the host via ssh):

1. Caddy unit + guard containers: systemctl --user is-active caddy.service; podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy' — expect Up … (healthy).
2. Anubis challenge page validity: curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143 — a healthy challenge contains "anubis_challenge" JSON with difficulty and the main JS.
3. Outpost errors: podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider' — routine dramatiq task logs are normal.
4. Post-auth handlers from INSIDE the caddy container (8016/8017 live in its netns — curl http://127.0.0.1:8017 from the HOST returns 000 connection refused, which is EXPECTED, not a fault): podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/' and the same for /services.json, /status.json. Expect 200 + body. (BusyBox wget in this container has NO -H flag — use --header "Host: …".)
5. Files on disk: ls -la /srv/homepage/ and validate status.json parses (python3 -c "import json;json.load(open('/srv/homepage/status.json'))").

Tool/behavior notes: the anubis container is distroless — podman exec anubis sh fails (sh: executable not found); inspect via its logs or the served challenge page. Anubis logs x-real-ip: 10.89.1.22 (the Caddy container's podman IP) for ALL clients — rootless-podman published-port SNAT; clients are indistinguishable by IP and clearance is cookie-based. Do NOT mistake that for the bug. Homepage render: app.js does Promise.all([fetch services.json, fetch status.json]); on ANY fetch failure the catch sets #summary to status unavailable: <err> and the grid stays EMPTY — which a user reads as "links gone / not loading." A 30-service status.json with a 9-service services.json renders fine (stateByName lookup).

Decision: if every layer is healthy (challenge valid, outpost clean, handlers 200, files render), the fault is client-side/session. STOP thrashing the live system — repeated failed browser challenges share the masked 10.89.1.22 identity. Get the authenticated view: ask for the exact symptom (challenge spinning? login loop? empty grid? "status unavailable" text?) or the authentik login to drive the page in a real browser.

Verify live

ssh dankclaw 'systemctl --user is-active anubis.service'
podman inspect anubis --format "{{.ImageName}} {{.State.Health.Status}}"
# the apex 308s to www; the guarded vhost is www — probe www, not the apex
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -A "Mozilla/5.0" https://franksdank.cloud/
# challenge: body carries the version marker + PoW page
curl -s -A "Mozilla/5.0" https://www.franksdank.cloud/ | grep -oE "cachebuster=v[0-9.]+|Making sure you.re not a bot"
# deny: HTTP 200 at v1.27+ — grep the body, not the status (gotcha 9)
curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && echo denied || echo BROKEN-challenge-served
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt