Redact a live credential that escaped into tracked memory

The auto-learned memory tier for serverAppConfigs carried, in plaintext, the
Caddy basicauth bcrypt hash for dankuser, the password behind it, and the
Pi-hole internal password. The file is tracked, so all three are in git
history on every machine that has pulled this repo, and in every agent
context window that has read the bucket.

Operational knowledge preserved in full (which auth layer fails, how to
reset, that the CLI hash is what CivetWeb validates, hot-reload through the
file watcher); only the credential values removed.

Both credentials still need rotation: redaction does not un-leak history.
That is an operator action on Dankclaw, not something this commit can do.

Refs #40

agent/memories/--home-dank-Projects-serverAppConfigs--/learned.md

@@ -7,9 +7,9 @@
 - Hermes (Dankclaw) model routing as of 2026-09-05: main default = frank-large (Dankclaw 192.168.0.143:7770/v1), small/aux routes = frank-small (Dankserver 192.168.0.235:7770/v1), vision + title_generation route to the exact FLM registry tag gemma4-it:e4b (192.168.0.143:7771/v1). The pre-rename aliases frank-default-large/-small/-sightsound are dead since the 2026-09-03 rename cutover — never route on them. Live config lives at /home/dank/Containers/hermes/data/config.yaml (owned by uid 109999; needs sudo to read/edit); the repo's dankclaw/hermes/config.yaml.example is a sanitized backup and can lag live, so diff live first before editing either. Signal adapter is currently disabled in live config (platforms.signal.enabled: false).
 - On the guarded franksdank.cloud vhosts (Caddy → Anubis :8923 → Authentik outpost), every unauthenticated request — roots and deep paths alike — is answered with HTTP **200** and an HTML challenge page (titleMaking sure you're not a bot!/title), not a 4xx. Verifying guard health by status code alone is meaningless; assert on leakage markers instead (copyparty UI title, homepage id="grid", services.json "category" JSON shape). Non-JS clients (curl/wget/WebDAV) can never solve the PoW challenge, so they are permanently parked at the challenge page — this is the entire security boundary for copyparty, which serves authenticated traffic with anonymous read perms (-v /w::r:ra,dank). Pinned by tests/e2e/specs/15-unauth-fetch.spec.ts (guest project, empty storage state): any guard-route regression that lets app bytes answer fails loudly; any gate implementation passes.
 - On Dankclaw's Pi-hole (FTL v6.7, quadlet at /home/dank/Containers/pihole), /etc/pihole/hosts/custom.list (host path /home/dank/Containers/pihole/etc-pihole/hosts/custom.list) is a LIVE-ONLY Custom-DNS source NOT tracked in the repo — it held ~22 A records pointing every *.franksdank.cloud vhost at 192.168.0.143, including names with no public Porkbun record. gravity.db's domainlist table has zero custom-DNS rows on this build, and the pihole CLI's sqlite path (pihole-FTL sqlite3) works but the web API auth was failing at discovery time, so retiring a name from LAN DNS requires editing custom.list directly (sudo sed -i '/name/d') followed by podman exec pihole pihole restartdns reload; deleting only the repo-managed addn-hosts record is not sufficient. Verify with dig +short name @192.168.0.143 → empty answer.
-- Pi-hole admin panel at pihole.franksdank.cloud has two auth layers: Caddy basicauth (edge, username dankuser + hash) and Pi-hole internal password (CivetWeb, set via pihole setpassword). The Pi-hole internal password was the actual failure point, not the Caddy hash. Pi-hole resolves to 192.168.0.143 via addn-hosts (all 28 names point here), making it a LAN service behind Caddy's public TLS edge. Options to remove public exposure: (1) remove the Caddy vhost entirely — LAN clients hit Pi-hole quadlet directly on :8083; (2) keep Caddy vhost but remove the Porkbun A record so external DNS won't resolve pihole.franksdank.cloud. Current state: Caddy basicauth hash = $2b$14$cJm.hbvDi1dEh7x0XRMnseaIdvIrRyH.AcTL56jGJf3wd54qvEb9K (dankuser, password pihole2026); Pi-hole internal password = pihole.2026 (set via pihole setpassword, survives restart).
+- Pi-hole admin panel at pihole.franksdank.cloud has two auth layers: Caddy basicauth (edge, username dankuser + hash) and Pi-hole internal password (CivetWeb, set via pihole setpassword). The Pi-hole internal password was the actual failure point, not the Caddy hash. Pi-hole resolves to 192.168.0.143 via addn-hosts (all 28 names point here), making it a LAN service behind Caddy's public TLS edge. Options to remove public exposure: (1) remove the Caddy vhost entirely — LAN clients hit Pi-hole quadlet directly on :8083; (2) keep Caddy vhost but remove the Porkbun A record so external DNS won't resolve pihole.franksdank.cloud. Current state: Caddy basicauth hash and the Pi-hole internal password are BOTH redacted (2026-10-20, issue #40) — they sat here in plaintext, which means they are in git history on every machine that ever pulled this repo, so both need rotation before this note is trustworthy; the live values are the Dankclaw host `.env` (`CADDY_BASICAUTH_HASH`) and `pihole setpassword`, never this file.
 - Pi-hole admin at pihole.franksdank.cloud has two auth layers: (1) Caddy basicauth edge (username dankuser, hash in rendered Caddyfile from CADDY_BASICAUTH_HASH in .env), (2) Pi-hole internal CivetWeb password (stored in cli_pw as bcrypt hash, set via pihole setpassword CLI). The env file WEBPASSWORD is used at container start but the CLI hash is what the running web server validates. When "I can't get into Pi-hole admin" returns 403 even after Caddy basicauth passes, the Pi-hole internal password is the likely culprit. Fix: podman exec pihole pihole setpassword newpw to update the CLI hash, then verify podman exec pihole cat /etc/pihole/cli_pw shows updated value. Verify by curling with basicauth: curl -sk -H 'Authorization: Basic ZGFua3VzZXI6...' https://pihole.franksdank.cloud/admin/ should return 302 (not 401 or 403).
-- Pi-hole admin panel at pihole.franksdank.cloud uses Caddy basicauth (not Pi-hole's own CivetWeb login). The hash is shared across services (RomM, Copyparty, Pi-hole) via CADDY_BASICAUTH_HASH in the host .env. The rendered Caddyfile at /home/dank/Containers/caddy/Caddyfile is mounted into the Caddy container — updates are picked up by Caddy's file watcher without restart. Password reset: generate bcrypt hash with python3 + bcrypt (hashpw + gensalt), update the rendered Caddyfile line directly or update .env + deploy. Caddy accepts both $2a$ and $2b$ bcrypt variants. _(context: Pi-hole admin password reset session: discovered Caddy basicauth (not Anubis), could not find matching password in common candidates, reset to pihole2026, updated rendered Caddyfile directly, synced repo .env files, Caddy file watcher handled hot-reload.)_
+- Pi-hole admin panel at pihole.franksdank.cloud uses Caddy basicauth (not Pi-hole's own CivetWeb login). The hash is shared across services (RomM, Copyparty, Pi-hole) via CADDY_BASICAUTH_HASH in the host .env. The rendered Caddyfile at /home/dank/Containers/caddy/Caddyfile is mounted into the Caddy container — updates are picked up by Caddy's file watcher without restart. Password reset: generate bcrypt hash with python3 + bcrypt (hashpw + gensalt), update the rendered Caddyfile line directly or update .env + deploy. Caddy accepts both $2a$ and $2b$ bcrypt variants. _(context: Pi-hole admin password reset session: discovered Caddy basicauth (not Anubis), could not find matching password in common candidates, reset to the current password [redacted 2026-10-20, issue #40], updated rendered Caddyfile directly, synced repo .env files, Caddy file watcher handled hot-reload.)_
 - On the FrankBook-Pro *arr fleet (current lsio images: Sonarr v4/Radarr v5 API v3, Lidarr v2/Readarr API v1), the download-client validation endpoints are NOT usable via API: POST /api/{v}/downloadclient/test/{id} and .../test?forceTest=true both return 405, and POST .../downloadclient/action/getTest exists but 400s without a bespoke body. To prove download-client health after changing a qB host/port, skip the API test route and prove behaviorally instead: podman exec arr curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/ (expect 200) plus a real qB login from inside the container (curl --data-urlencode username=admin --data-urlencode password=$QBIT_PW .../api/v2/auth/login, expect 200), then confirm /api/{v}/health shows no DownloadClient-*/wbrrelated warnings. Also: arr APIs redact stored client passwords as ******** (sonarr/radarr/lidarr) — PUT the object back with the sentinel untouched to preserve the stored value; readarr (v1) returns the plaintext. When editing via the edit tool, body rows must be prefixed with +, not : — : prefixes get written literally into the file. _(context: Capture turn after the arr→qB loopback repoint (2026-09-04): app-side download-client test endpoints were unreachable, so client health was proven behaviorally instead.)_
 - The *arr fleet (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) runs ONLY on FrankBook-Pro (192.168.0.112, Mac podman machine) since 2026-09-04 — the dankserver compose copy was removed (config roots in ~/.AGENTTRASH/20260904-dankserver-arr-teardown, 22 MB, awaiting purge approval). Fleet invariant, enforced by tests/frankbook_quadlet.bats: ALL six containers share gluetun's netns (Network=gluetun.container, Wants/After gluetun.service) and publish NO ports; the gluetun unit publishes the union (8080, 6881 tcp+udp, 9696, 8989, 7878, 8686, 8787), so all egress exits ProtonVPN by construction (AllowedIPs=0.0.0.0/0; proven: netns egress 51.179.177.14 vs residential 71.227.227.172). deploy-plan.py couples gluetun → all six joiners (a gluetun recreate kills the whole family). Consequences: (1) the old arr→qB addressing hack 10.88.0.1:8080 (podman-bridge gateway) routes INTO the tunnel from inside the netns — download clients must use 127.0.0.1:8080 (done via arr APIs, verified by real qB login from inside sonarr's container); same rule for any future Prowlarr Application rows (baseUrl 127.0.0.1:arr port). (2) Caddy arr vhosts and homepage probes target 192.168.0.112:port (host-side published ports unaffected by netns changes). (3) Known defects as of 2026-09-04: Prowlarr's Applications table is EMPTY (the four rows from 2026-08-30 were lost — prowlarr→arr sync broken until rebuilt); readarr's qB proxy auth defect needs retest post-move; EZTV/1337x "CloudFlare-blocked" parking is stale (egress now VPN — retest before enabling). (4) E2E lesson: playwright specs 11-admin/12-crawl now derive expected open-link counts from services.json (showLink) instead of hardcoded counts — keep new registry entries consistent. Also: deploy.sh --service X dispatches ONLY X on frankbook (path-coupling applies to changed-path planning); gluetun restart self-heals all netns joiners via Restart=always so a single gluetun deploy converges the family. _(context: 2026-09-04 operator decision session: dankserver arr compose copy (booted that morning) removed after Frank chose FrankBook-Pro as the sole arr deployment; netns consolidation + test seam + loopback repoint executed same session. Commits 4c7465d, df4339a.)_
 - Rootless podman on Dankserver (5.7.0, verified 2026-09-04 via /proc/pid/uid_map across 7 containers): the DEFAULT user namespace maps container uid 0 → host uid 1000 (dank) and container uid N → host 100000+N-1 (e.g. calibre-web PUID=1000 writes as 100999). Consequences: (1) PUID=0 + data roots owned 1000:1000 is the complete fleet recipe — NO UserNS/userns_mode key is needed on quadlets or compose services; the plex quadlet works without one. (2) NEVER add userns_mode: host to a podman-compose service: podman-compose 1.5.0 runs every compose service inside a pod and podman hard-rejects the combo (--userns and --pod cannot be set together), leaving the stack down with no recreate path. (3) Diagnostics: podman exec c id always shows uid=0 inside any mapping — meaningless; ground truth is /proc/container-pid/uid_map, host-side ps -o user= -p pid, or ownership of freshly-written files. (4) *arr boot-crash signature: UnauthorizedAccessException: Access to the path '/config/...' at logger init = data root not writable (e.g. root-owned dir created by sudo mkdir); the app exits before binding its port, so the port probe reads "dead" rather than "crashed" — check podman logs tail for "Press enter to exit" before assuming an outage has an infra cause. (5) NEVER pkill -f rootlessport on these hosts: it kills the port-forward helper of EVERY published container simultaneously (2026-09-03 outage); restart the owning systemd units instead. _(context: Dankserver *arr stack restore session, 2026-09-04: stack committed 2026-08-29 never booted (Sentry crash, dead ports read as outage); userns_mode: host fix was a red herring — rejected by podman-compose pods; real fix was chown 1000:1000 of /opt config dirs. Commit e454159 (revert), 8a63624 (docs).)_