serverAppConfigs/serverAppConfigs-docs/Infrastructure/Anubis.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	Anubis is the bot-challenge proxy that fronts the guarded public vhosts on	infrastructure anubis	2026-09-01

Anubis — bot challenge proxy

Anubis is the bot-challenge proxy that fronts the guarded public vhosts on Dankclaw. Browsers get a proof-of-work challenge; AI scraper user-agents are denied outright. It sits in a two-tier chain: client → Caddy TLS listener → Anubis (:8923, internal-only) → Caddy internal listener (:8016) → app. Host headers survive both hops, so the Authentik embedded outpost can route on the original Host behind it. Since v1.27.0 the challenge cookie is domain-wide (COOKIE_DOMAIN=franksdank.cloud): a browser solves the proof-of-work once on any guarded vhost and every guarded *.franksdank.cloud vhost honors it until the 168h expiry or an authentik re-login. See Caddy and Authentik.

graph LR
    C[Client] --> T[Caddy TLS listener]
    T --> A[Anubis verify :8923]
    A -->|accepted| I[Caddy internal listener :8016]
    A -->|challenge or deny| C
    I --> P[whole-request proxy to Authentik outpost]

Facts (live inventory)

Item	Value
Runtime	Quadlet unit anubis.service (dankclaw/anubis/quadlet/anubis.container) — boot-persistent, self-healing
Image	ghcr.io/techarohq/anubis:v1.27.0, run under a localhost/ alias tag (see gotcha 1)
Ports	:8923 internal (no published port — podman ps shows no port mapping, 2026-10-18); metrics on :9090 reachable only inside dankclaw_default (dankclaw/anubis/quadlet/anubis.container:3,58-61)
Config surface	flags via flagenv — TARGET=http://caddy:8016 sets --target, POLICY_FNAME=/policy/policy.json sets --policy-fname, COOKIE_DOMAIN=franksdank.cloud sets --cookie-domain
Policy	dankclaw/anubis/policy.json mounted read-only into the container (anubis.container:52); parsed at process start only — the deployer restarts the unit after every policy change (scripts/deploy-remote.sh:1091) and no reload is wired here (unverified 2026-10-18: whether upstream Anubis answers any reload signal at all; upstream docs are themselves behind an Anubis challenge, so this could not be read)
Store	memory JWTs signed by a persistent ED25519 key (ED25519_PRIVATE_KEY_HEX, host file /home/dank/Containers/anubis/env, mode 0600, gitea-style ${VAR} expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot

Policy content, in the file's own order — first match wins, so the DENY really does come first (dankclaw/anubis/policy.json, 2026-10-18): a DENY for AI-scraper user-agent regexes (GPTBot, ChatGPT-User, OAI-SearchBot, Claude-*, anthropic-ai, Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, Google-Extended, DuckAssistBot, cohere-ai, YouBot, KimiBot, Applebot-Extended; lines 2-7), then an ALLOW for Anubis's own endpoint namespace ^/\.within\.website (lines 8-12 — the challenge JS calls make-challenge/pass-challenge under that prefix, and challenging those endpoints makes the challenge challenge itself), then an ALLOW for the Authentik transport prefix ^/outpost\.goauthentik\.io/ (lines 13-17 — login, start, callback and the outpost's static assets must be reachable before any session or challenge dance completes; authentik's own session gate protects them), then CHALLENGE for everything else (path_regex: "." catch-all, lines 18-22). DNSBL off (dnsbl: false, line 23). Live on 2026-10-18: a Mozilla/5.0 request to https://www.franksdank.cloud/ returns 200 carrying anubis_challenge and the cachebuster=v1.27.0 marker, while -A GPTBot returns the deny body — see gotcha 9 for why that is also a 200.

Valid action verbs in this file are ALLOW, CHALLENGE, DENY — ACCEPT is not one of them (first deploy of that rule crash-looped the unit on exactly that typo; verified against the image binary's string table). The local policy uses only the JSON rule form. Upstream documents the policy file as YAML with JSON accepted because it is valid YAML "as of version v1.17.0 or later" (https://anubis.techaro.lol/docs/admin/policies/), and import statements exist in the 1.x line (https://github.com/TecharoHQ/anubis/releases lists "allow wildcard matches in import statements" on the road to v1.27.0) — so the older local statement "JSON-only, no (data)/ imports, no CEL at this tag" is a v1.13-era claim kept here as history and is not a property of v1.27.0; whether CEL expressions are accepted at this exact tag was not verifiable (unverified 2026-10-18). What is verified is that this JSON policy loads and is enforced (container healthy; 136 challenges issued in the retained log since the 01:25 boot, scraper deny confirmed by probe). Upgrading the pin = pull + re-tag the localhost/ alias, restart; verify the flag surface with podman run --rm ... --help and the policy against the new tag first (done for v1.27.0). Upstream is ahead of the pin: Anubis's own documentation site served its challenge page with cacheBuster=v1.28.2-0.20261012194510-… when fetched on 2026-10-18, so v1.28.x exists.

Exempt by construction

These never route through Anubis (no policy bypass rule needed): auth. (the IdP login itself must stay ungated, dankclaw/Caddyfile:90-99), gitea (git smart-HTTP clients cannot solve JS challenges — /info/refs and git-*-pack paths pass unchallenged, dankclaw/Caddyfile:110-120), ai. OpenWebUI (own SSO + API-key auth, dankclaw/Caddyfile:145-151), pihole. (a deliberate basicauth edge instead, dankclaw/Caddyfile:162-171), hermes. and hermesadmin. (app-enforced passwords, no public DNS for the first, dankclaw/Caddyfile:179-201), plex. including the plain-HTTP legacy name (dankclaw/Caddyfile:203-238), the media vhosts books./jelly. (dankclaw/Caddyfile:241-251), and the FrankBook-Pro *arr/qBittorrent vhosts (dankclaw/Caddyfile:260-352). There is also no vhost at all for ComfyUI, so it is exempt by absence — and therefore unauthenticated on the LAN at comfyui.franksdank.cloud:8188. The guarded set is exactly www., search., grafana., prometheus., loki., seerr., marreta., romm., files. (dankclaw/Caddyfile:283-423).

Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)

1. Image ref casing. Podman's reference parser rejects mixed-case repository refs before any network I/O (repository name must be lowercase). All refs use the lowercase ghcr.io/techarohq/anubis form — ghcr resolves case-insensitively at pull time.

2. Quadlet + registry-prefixed refs. On this host's podman 5.7 quadlet path, a container created from a registry-prefixed image ref fails with image not known even when the image is present in storage; the identical argv works from a shell or a transient unit. The unit therefore runs the image under a localhost/techarohq/anubis:v1.27.0 alias tag — deploy pulls the canonical ref, then podman tags the alias (idempotent; scripts/deploy-remote.sh:1078-1084, and podman inspect anubis reports localhost/techarohq/anubis:v1.27.0 on 2026-10-18). The pin used to be v1.13.0; that name is retired, and the unit sets Pull=never (anubis.container:54-56) so a boot cannot stall on ghcr rate limits.

3. Health probe exec form. The image is a distroless ko-build (no shell; entrypoint /ko-app/anubis). This quadlet generator joins HealthCmd into one \x20-escaped argv element and podman then runs it shell-form — exec of a missing shell exits 1 and the container sits unhealthy. A JSON exec-form array (HealthCmd=["/ko-app/anubis","--healthcheck"]) passes through the generator verbatim and executes the binary directly.

4. Rollback nuance. systemctl --user stop anubis.service makes every guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not forward_auth — Caddy's public listeners proxy to :8923). A true rollback is a Caddyfile revert + redeploy of the caddy service.

5. Deploy handler pull guard. deploy_anubis (scripts/deploy-remote.sh:1074-1092) guards the pull with podman image exists ghcr.io/techarohq/anubis:v1.27.0; a cold anonymous ghcr token request can 403 on the first fetch — retry once, the auth cache makes the second attempt local. The handler always ends with systemctl --user restart anubis.service + a health wait, which is what makes a policy change land (gotcha 7).

6. JS-capable clients are the canary. Non-JS clients (curl, bots, scrapers) never solve challenges — they just receive the challenge page, so every curl probe looks green while the verification step is broken. The first JS-capable client to traverse a freshly deployed chain is the first real test of it: browsers POST-solve via make-challenge and GET-navigate pass-challenge?response=<sha256(challenge+nonce)>&nonce=N &elapsedTime=T&redir=<path>; if any own-endpoint path is challenged instead of allowed, the browser's fetch throws and it renders its own error page (iOS/macOS browsers phrase it as "the request failed and the interceptors did not return an alternative response") instead of the login flow. Verify with a JS-capable client end-to-end — mint via make-challenge, solve, submit with browser-exact query params — before declaring a challenge chain healthy.

7. Single-file bind mount + git pull = stale policy. The policy is mounted file-at-a-time (/home/dank/serverAppConfigs/dankclaw/anubis/ policy.json:/policy/policy.json:ro); bind mounts pin the INODE. A git pull/checkout that replaces the file leaves the running container reading the old inode indefinitely (observed live: 22 h of uptime serving a policy already superseded on disk, with no reload log line because there is nothing watching). A policy change only takes effect when the anubis container is (re)started — ./scripts/deploy.sh dankclaw --service anubis — and the deployer must therefore always restart it rather than trust the file contents on disk.

8. X-Real-Ip is mandatory at v1.27+ (HTTP 500 without it). Anubis now answers any request lacking X-Real-Ip with a 500 whose log line reads [misconfiguration] X-Real-Ip header is not set. The real chain always sets it (every guarded Caddy listener carries header_up X-Real-Ip {remote_host} — the bats seam enforces the line), so users never see this. But any direct probe of :8923 (scratch container, curl, a new reverse-proxy hop) MUST set the header or it looks like a policy failure. Scratch-verify with -H "X-Real-Ip: 192.168.0.20".

9. DENY renders as an error-styled page at HTTP 200. At v1.27 the denied client gets the "Oh noes!" template containing Access Denied and error code <rule-id> — with status 200, not 403 (the boot log maps rule names to error ids: rule-error-ids). Status-only probes cannot distinguish challenge from deny at this release; grep the body for Access Denied / Making sure you instead.

Guard-chain layer diagnosis (a user reports a guarded page "broken")

Use when a guarded page (e.g. the www.franksdank.cloud homepage) reads as "broken / links gone / not loading" and the fault must be localized to server-side or client-side/session. Chain topology: Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home/@data handlers) → app/static.

Layer-by-layer checks (run from the host via ssh):

1. Caddy unit + guard containers: systemctl --user is-active caddy.service; podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy' — expect Up … (healthy).
2. Anubis challenge page validity: curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143 — a healthy challenge contains "anubis_challenge" JSON with difficulty and the main JS.
3. Outpost errors: podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider' — routine dramatiq task logs are normal.
4. Post-auth handlers from INSIDE the caddy container (8016/8017 live in its netns — curl http://127.0.0.1:8017 from the HOST returns 000 connection refused, which is EXPECTED, not a fault): podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/' and the same for /services.json, /status.json. Expect 200 + body. (BusyBox wget in this container has NO -H flag — use --header "Host: …".)
5. Files on disk: ls -la /srv/homepage/ and validate status.json parses (python3 -c "import json;json.load(open('/srv/homepage/status.json'))").

Tool/behavior notes: the anubis container is distroless — podman exec anubis sh fails (sh: executable not found); inspect via its logs or the served challenge page. Anubis logs the Caddy container's podman address as x-real-ip for ALL clients — rootless-podman published-port SNAT; clients are indistinguishable by IP and clearance is cookie-based. Do NOT mistake that for the bug. The address is whatever netavark gave Caddy on dankclaw_default — all 138 logged x-real-ip values in the container's retained log read 10.89.1.8 (2026-10-18), and it is not a pinned address, so never fingerprint "the Caddy hop" on a literal 10.89.1.22-style value. Homepage render: app.js does Promise.all([fetch services.json, fetch status.json]); on ANY fetch failure the catch sets #summary to status unavailable: <err> and the grid stays EMPTY — which a user reads as "links gone / not loading." A status.json that carries states for more services than services.json lists renders fine (stateByName lookup); today both files carry 35 entries (shared/homepage/services.json, /srv/homepage/status.json, 2026-10-18).

Decision: if every layer is healthy (challenge valid, outpost clean, handlers 200, files render), the fault is client-side/session. STOP thrashing the live system — repeated failed browser challenges share the masked 10.89.1.22 identity. Get the authenticated view: ask for the exact symptom (challenge spinning? login loop? empty grid? "status unavailable" text?) or the authentik login to drive the page in a real browser.

Verify live

ssh dankclaw 'systemctl --user is-active anubis.service'
podman inspect anubis --format "{{.ImageName}} {{.State.Health.Status}}"
# the apex 308s to www; the guarded vhost is www — probe www, not the apex
curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" -A "Mozilla/5.0" https://franksdank.cloud/
# challenge: body carries the version marker + PoW page
curl -s -A "Mozilla/5.0" https://www.franksdank.cloud/ | grep -oE "cachebuster=v[0-9.]+|Making sure you.re not a bot"
# deny: HTTP 200 at v1.27+ — grep the body, not the status (gotcha 9)
curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && echo denied || echo BROKEN-challenge-served
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt

Ran on 2026-10-18 from Dankclaw with --resolve …:443:192.168.0.143: the apex 308s to https://www.franksdank.cloud/; www. returns 200 with anubis_challenge + cachebuster=v1.27.0; -A GPTBot returns 200 whose body contains Access Denied; search./outpost.goauthentik.io/login returns 302 (exempt, not challenged). systemctl --user is-active anubis.service → active, is-enabled → generated, container health healthy.