Frank
4b82f35354
Tracked the harness artifacts accumulated on blankpad (blank@), matching the existing convention for the --home-dank-* memory stores and the 36 already-tracked managed-skills. managed-skills (9): authentik-proxy-provider-app-link, dankclaw-guard-chain-layer-diagnosis, e2e-arr-rawip-reachability-triage, halogen-kv-pool-fit, lan-client-split-dns-fix, noctalia-reference, rootless-quadlet-log-retrieval, ssh-key-bootstrap-no-sshpass, swapfile-resize-safe-ordering. memories (2 stores): --home-blank-.omp-- (dotfiles_git, handy_lifecycle, input_permissions) and --home-blank-Projects-serverAppConfig-- (frank_large_quadlet). Secret sweep before staging: no PEM blocks, no credential literals. Content is fleet topology/IPs only, same class as the tracked dank stores.
34 lines
3.5 KiB
Markdown
34 lines
3.5 KiB
Markdown
---
|
|
name: dankclaw-guard-chain-layer-diagnosis
|
|
description: "When a user reports a guarded dankclaw page (homepage/vhost) broken but the server may be healthy: verify each layer of the Anubis→Authentik→Caddy chain server-side and isolate client-side vs server-side."
|
|
---
|
|
|
|
# Dankclaw Guard-Chain Layer Diagnosis
|
|
|
|
Use when a user reports a guarded page (e.g. `www.franksdank.cloud` homepage) "broken / links gone / not loading" and you must determine whether the fault is server-side or client-side/session.
|
|
|
|
## Chain topology
|
|
`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost → Caddy :8017 (@home / @data handlers) → app/static`
|
|
- Public vhosts `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`.
|
|
- Post-auth `http://:8017` listener routes by `@<svc> host <vhost>`.
|
|
- Homepage static: `/srv/homepage/` (index.html, app.js, style.css, services.json, status.json, deployed.js).
|
|
|
|
## Layer-by-layer checks (run from the host via ssh)
|
|
1. **Caddy unit + guard containers**: `systemctl --user is-active caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE 'anubis|authentik|caddy'` — expect `Up ... (healthy)`.
|
|
2. **Anubis challenge page validity**: `curl -sk https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143` — a healthy challenge contains `"anubis_challenge"` JSON with `difficulty` and the main JS. If present, the guard serves fine.
|
|
3. **Authentik outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep -iE 'error|exception|traceback|500|provider'` — routine dramatiq task logs are normal.
|
|
4. **@home / @data from INSIDE the caddy container** (the only way to reach 8016/8017):
|
|
`podman exec caddy sh -c 'wget -q -S -T5 --header "Host: www.franksdank.cloud" -O- http://127.0.0.1:8017/'`
|
|
and the same for `/services.json`, `/status.json`. Expect 200 + body.
|
|
5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json` parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
|
|
|
|
## Gotchas (cost me calls)
|
|
- **BusyBox wget in the caddy container**: NO `-H` flag. Use `--header "Host: ..."`.
|
|
- **8016/8017 live in the caddy container's netns**: `curl http://127.0.0.1:8017` from the HOST returns `000` (connection refused) — EXPECTED, not a fault. Test from inside the container.
|
|
- **anubis container is distroless**: `podman exec anubis sh` fails (`sh: executable not found`). Inspect via its logs / the served challenge page instead.
|
|
- **x-real-ip masking**: Anubis logs `x-real-ip: 10.89.1.22` (Caddy container podman IP) for ALL clients — rootless-podman published-port SNAT. Pre-existing; clients indistinguishable by IP; clearance is cookie-based. Do NOT mistake this for the bug.
|
|
- **Homepage render**: `app.js` does `Promise.all([fetch services.json, fetch status.json])`; on ANY fetch failure the catch sets `#summary` to `status unavailable: <err>` and the grid stays EMPTY — which a user reads as "links gone / not loading." A 30-service status.json with a 9-service services.json renders fine (stateByName lookup).
|
|
|
|
## Decision
|
|
If every layer above is healthy (challenge valid, outpost clean, @home/@data 200, files render), the fault is client-side/session. STOP thrashing the live system (repeated failed browser challenges share the masked 10.89.1.22 identity). Get the authenticated view: ask for the exact symptom (challenge spinning? Authentik login loop? empty grid? "status unavailable" text?) or the Authentik login to drive the page in a real browser.
|