dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Infrastructure/Anubis.md
+84 -34
View File
@@ -35,37 +35,67 @@ graph LR
|------|-------|
| Runtime | Quadlet unit `anubis.service` (`dankclaw/anubis/quadlet/anubis.container`) — boot-persistent, self-healing |
| Image | `ghcr.io/techarohq/anubis:v1.27.0`, run under a `localhost/` alias tag (see gotcha 1) |
| Ports | `:8923` internal (no published port); metrics on `:9090` reachable only inside `dankclaw_default` |
| Ports | `:8923` internal (no published port — `podman ps` shows no port mapping, 2026-10-18); metrics on `:9090` reachable only inside `dankclaw_default` (`dankclaw/anubis/quadlet/anubis.container:3,58-61`) |
| Config surface | flags via flagenv — `TARGET=http://caddy:8016` sets `--target`, `POLICY_FNAME=/policy/policy.json` sets `--policy-fname`, `COOKIE_DOMAIN=franksdank.cloud` sets `--cookie-domain` |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart) |
| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container (`anubis.container:52`); parsed at process start only — the deployer restarts the unit after every policy change (`scripts/deploy-remote.sh:1091`) and no reload is wired here *(unverified 2026-10-18: whether upstream Anubis answers any reload signal at all; upstream docs are themselves behind an Anubis challenge, so this could not be read)* |
| Store | memory JWTs signed by a **persistent** ED25519 key (`ED25519_PRIVATE_KEY_HEX`, host file `/home/dank/Containers/anubis/env`, mode 0600, gitea-style `${VAR}` expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot |
Policy content: an `ALLOW` rule for Anubis's own endpoints (the
`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
those endpoints makes the challenge challenge itself), an `ALLOW` rule for
the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
callback and the outpost's static assets must be reachable before any
session or challenge dance completes — authentik's own session gate protects
them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
Valid action verbs
at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
(first deploy of that rule crash-looped the unit on exactly that typo;
verified against the image binary's string table). The policy schema at this
tag is JSON-only — no `(data)/` imports and no CEL; first match wins (JSON
still loads at v1.27.0 — scratch-verified 2026-09-02 before the pin bump).
Policy content, in the file's own order — first match wins, so the DENY
really does come first (`dankclaw/anubis/policy.json`, 2026-10-18): a **DENY**
for AI-scraper user-agent regexes (GPTBot, ChatGPT-User, OAI-SearchBot,
Claude-\*, anthropic-ai, Perplexity\*, CCBot, Bytespider, Amazonbot,
meta-externalagent, Google-Extended, DuckAssistBot, cohere-ai, YouBot,
KimiBot, Applebot-Extended; lines 2-7), then an **ALLOW** for Anubis's own
endpoint namespace `^/\.within\.website` (lines 8-12 — the challenge JS calls
`make-challenge`/`pass-challenge` under that prefix, and challenging those
endpoints makes the challenge challenge itself), then an **ALLOW** for the
Authentik transport prefix `^/outpost\.goauthentik\.io/` (lines 13-17 — login,
start, callback and the outpost's static assets must be reachable before any
session or challenge dance completes; authentik's own session gate protects
them), then **CHALLENGE** for everything else (`path_regex: "."` catch-all,
lines 18-22). DNSBL off (`dnsbl: false`, line 23). Live on 2026-10-18: a
`Mozilla/5.0` request to `https://www.franksdank.cloud/` returns 200 carrying
`anubis_challenge` and the `cachebuster=v1.27.0` marker, while
`-A GPTBot` returns the deny body — see gotcha 9 for why that is also a 200.
Valid action verbs in this file are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT`
is not one of them (first deploy of that rule crash-looped the unit on
exactly that typo; verified against the image binary's string table). The
local policy uses only the JSON rule form. Upstream documents the policy file
as YAML with JSON accepted because it is valid YAML "as of version v1.17.0 or
later" (https://anubis.techaro.lol/docs/admin/policies/), and import
statements exist in the 1.x line (https://github.com/TecharoHQ/anubis/releases
lists "allow wildcard matches in import statements" on the road to v1.27.0) —
so the older local statement "JSON-only, no `(data)/` imports, no CEL at this
tag" is a v1.13-era claim kept here as history and is **not** a property of
v1.27.0; whether CEL expressions are accepted at this exact tag was not
verifiable *(unverified 2026-10-18)*. What is verified is that this JSON
policy loads and is enforced (container `healthy`; 136 challenges issued in the
retained log since the 01:25 boot, scraper deny confirmed by probe).
Upgrading the pin = pull + re-tag the `localhost/` alias, restart; verify
the flag surface with `podman run --rm ... --help` and the policy against
the new tag first (done for v1.27.0).
the new tag first (done for v1.27.0). Upstream is ahead of the pin: Anubis's
own documentation site served its challenge page with
`cacheBuster=v1.28.2-0.20261012194510-…` when fetched on 2026-10-18, so
v1.28.x exists.
## Exempt by construction
These never route through Anubis (no policy bypass rule needed): plex
(plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay
ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
`/info/refs` and `git-*-pack` paths pass unchallenged).
These never route through Anubis (no policy bypass rule needed): `auth.`
(the IdP login itself must stay ungated, `dankclaw/Caddyfile:90-99`), `gitea`
(git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
`git-*-pack` paths pass unchallenged, `dankclaw/Caddyfile:110-120`), `ai.`
OpenWebUI (own SSO + API-key auth, `dankclaw/Caddyfile:145-151`), `pihole.`
(a deliberate basicauth edge instead, `dankclaw/Caddyfile:162-171`), `hermes.`
and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
`dankclaw/Caddyfile:179-201`), `plex.` including the plain-HTTP legacy name
(`dankclaw/Caddyfile:203-238`), the media vhosts `books.`/`jelly.`
(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
it is exempt by absence — and therefore unauthenticated on the LAN at
`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
(`dankclaw/Caddyfile:283-423`).
## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)
@@ -77,8 +107,12 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
path, a container created from a registry-prefixed image ref fails with
`image not known` even when the image is present in storage; the identical
argv works from a shell or a transient unit. The unit therefore runs the
image under a `localhost/techarohq/anubis:v1.13.0` alias tag — deploy
pulls the canonical ref, then `podman tag`s the alias (idempotent).
image under a `localhost/techarohq/anubis:v1.27.0` alias tag — deploy
pulls the canonical ref, then `podman tag`s the alias (idempotent;
`scripts/deploy-remote.sh:1078-1084`, and `podman inspect anubis` reports
`localhost/techarohq/anubis:v1.27.0` on 2026-10-18). The pin used to be
v1.13.0; that name is retired, and the unit sets `Pull=never`
(`anubis.container:54-56`) so a boot cannot stall on ghcr rate limits.
3. **Health probe exec form.** The image is a distroless ko-build (no shell;
entrypoint `/ko-app/anubis`). This quadlet generator joins `HealthCmd`
into one `\x20`-escaped argv element and podman then runs it shell-form —
@@ -89,9 +123,13 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not
forward_auth — Caddy's public listeners proxy to `:8923`). A true rollback
is a Caddyfile revert + redeploy of the caddy service.
5. **Deploy handler pull guard.** `deploy_anubis` guards the pull with
`podman image exists`; a cold anonymous ghcr token request can 403 on the
first fetch — retry once, the auth cache makes the second attempt local.
5. **Deploy handler pull guard.** `deploy_anubis`
(`scripts/deploy-remote.sh:1074-1092`) guards the pull with
`podman image exists ghcr.io/techarohq/anubis:v1.27.0`; a cold anonymous
ghcr token request can 403 on the first fetch — retry once, the auth cache
makes the second attempt local. The handler always ends with
`systemctl --user restart anubis.service` + a health wait, which is what
makes a policy change land (gotcha 7).
6. **JS-capable clients are the canary.** Non-JS clients (curl, bots,
scrapers) never solve challenges — they just receive the challenge page,
so every curl probe looks green while the verification step is broken.
@@ -163,15 +201,20 @@ Layer-by-layer checks (run from the host via ssh):
Tool/behavior notes: the anubis container is distroless — `podman exec anubis
sh` fails (`sh: executable not found`); inspect via its logs or the served
challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
podman IP) for ALL clients — rootless-podman published-port SNAT;
clients are indistinguishable by IP and clearance is cookie-based. Do NOT
mistake that for the bug. Homepage render: `app.js` does
challenge page. Anubis logs the Caddy container's podman address as
`x-real-ip` for ALL clients — rootless-podman published-port SNAT; clients
are indistinguishable by IP and clearance is cookie-based. Do NOT mistake
that for the bug. The address is whatever netavark gave Caddy on
`dankclaw_default` — all 138 logged `x-real-ip` values in the container's
retained log read `10.89.1.8` (2026-10-18), and it is not a pinned address,
so never fingerprint "the Caddy hop" on a literal `10.89.1.22`-style value.
Homepage render: `app.js` does
`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
failure the catch sets `#summary` to `status unavailable: <err>` and the
grid stays EMPTY — which a user reads as "links gone / not loading." A
30-service status.json with a 9-service services.json renders fine
(stateByName lookup).
status.json that carries states for more services than services.json lists
renders fine (stateByName lookup); today both files carry 35 entries
(`shared/homepage/services.json`, `/srv/homepage/status.json`, 2026-10-18).
**Decision:** if every layer is healthy (challenge valid, outpost clean,
handlers 200, files render), the fault is client-side/session. STOP
@@ -194,3 +237,10 @@ curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && e
# outpost transport exempt (ALLOW rule)
curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
```
Ran on 2026-10-18 from Dankclaw with `--resolve …:443:192.168.0.143`: the apex
308s to `https://www.franksdank.cloud/`; `www.` returns 200 with
`anubis_challenge` + `cachebuster=v1.27.0`; `-A GPTBot` returns 200 whose body
contains `Access Denied`; `search./outpost.goauthentik.io/login` returns 302
(exempt, not challenged). `systemctl --user is-active anubis.service` →
`active`, `is-enabled` → `generated`, container health `healthy`.