serverAppConfigs/serverAppConfigs-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Bundle Change Log	Chronological record of meaningful changes to the serverAppConfigs knowledge base.	changelog documentation maintenance	2026-09-14

Bundle Change Log

2026-08-30 (late) — download chain proven end to end; qB credentials persisted

• Cracked the arr↔Prowlarr feed contract: route is /{indexerId}/api (Prowlarr NewznabController; absolute /api/v1/indexer/{id}/newznab), auth = the Prowlarr global key. The previously hand-rolled /torznab/{id}/feed paths were Forms-auth catch-alls returning 302 → /login — which the arrs XML-parsed as "unexpected token '/'". A source-verified librarian answer settled it in one shot.
• Re-pointed all four arr indexer rows (Radarr→YTS id7, Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API; every PUT passed the app's live connection validator (202); RSS enabled.
• TV + music chains proven by live search: Sonarr returned 76 current releases via Prowlarr-Nyaa; Lidarr initially logged 0 active indexers (new rows default all-off) — enabling the search flags yielded 18 releases for YOASOBI – THE BOOK.
• First real grab: MoviesSearch → 3 YTS releases → The Dark Knight (2008) 2160p (7.5 GiB) → qBittorrent 100 % → imported (hasFile: true, 8,078,103,833 B at /movies/The Dark Knight (2008), hardlinked from the shared /mnt/entropy filesystem), queue empty.
• Prowlarr Application auto-sync found broken by version skew (Unable to cast object of type 'NzbDrone.Core.Thing…' against Radarr 6.3) — the manual rows are the integration; documented.
• qBittorrent permanent credentials installed: the WebUI temp password rotated on every boot (it had silently broken the arr clients once this session). qb5 removed the API password endpoint and the lscr image has no env support — wrote WebUI\Password_PBKDF2 (source-verified base64(salt):base64(PBKDF2-HMAC-SHA512/100k) form) into the conf with the container stopped; login survives podman restart (204). All arr download clients updated to it; value kept host-local in /home/dank/.qbpass (0600), never in git.
• Readarr parked with reason: abandoned upstream; its qB client cannot authenticate against qB 5 while three sibling apps pass with identical credentials. Indexer row healthy; books pipeline awaits a successor.
• Docs: rewrote Dankserver/prowlarr.md (live wiring + six verified gotchas incl. the joiner bug and the 302-masquerading-as-auth failure) and Dankserver/qbittorrent.md (persistent-credential recipe, Referer CSRF note).

2026-08-29 (late) — guarded-host sign-in fixed end to end

• Diagnosed both user symptoms: Invalid next URL = the flow being entered with an absolute cross-host next= (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had no auth A record (added via Porkbun API, auth A 71.227.227.172 TTL 300).
• Disproved meta-header routing live (full meta-header set still 404s; matching Host alone answers). The 2026.8 embedded outpost routes solely on the original Host vs each proxy provider's external_host.
• Provisioned proxy providers + applications for search, hermes, prometheus, romm, files (pks 2–6) via ORM, attached to the embedded outpost; internal_host = http://caddy:8017 everywhere.
• Wedge found + fixed live: providers created with null authorization_flow/invalidation_flow starve the outpost's config fetch (Failed to fetch outpost from API, retrying in 3 seconds forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
• /dankclaw/Caddyfile (29f9a86) — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; deleted the per-vhost redirect/flow design and stale comments.
• /dankclaw/anubis/policy.json (2a72efa) — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance; app paths remain challenged.
• Gotcha: the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile). Suite green (0 not-ok).
• /docs/Infrastructure/Authentik.md, /docs/Infrastructure/Anubis.md, /docs/Architecture/Network Architecture.md (ee1e038) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
• Verified by headless JS client: https://search.franksdank.cloud/ → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 state JWT. Credential submission is the user's step.

2026-09-14

• Creation: Hermes-WebUI — new hermes-webui compose service on Dankclaw (the Hermex server): runs hermes-webui's bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data with the gateway (identical rootless Podman uid mapping), publishes 127.0.0.1:8787 only, password-gated via HERMES_WEBUI_PASSWORD in the host .env.
• /dankclaw/compose.yaml, /dankclaw/.env.example — service definition and password placeholder; container-internal bind is 0.0.0.0 (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
• /tests/hermes_webui.bats — new seam: foreground entrypoint, loopback publish + auth, shared volume, /health healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
• Remote access pattern documented: iPhone Termius local forward 127.0.0.1:8787 -> 127.0.0.1:8787 over the existing router port forward to Dankclaw :22; Hermex points at http://localhost:8787. No Tailscale, no Cloudflare.

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts now proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
• /docs/Infrastructure/Caddy.md — Documented the Plex TLS-upstream requirement and the /web/ vs / distinction.
• /tests/caddyfile.bats — Added a test pinning the TLS upstream transport on both Plex vhosts.
• /docs/Dankclaw/Gitea.md — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
• /docs/Dankclaw/Dankclaw Stack.md — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
• /docs/Architecture/Deployment Flow.md — Replaced the split-proxy safety description with the canonical Caddy ownership model.
• /dankclaw/gitea/quadlet/gitea-caddy.container — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
• /scripts/deploy-plan.py — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
• /scripts/deploy.sh and /scripts/deploy-remote.sh — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
• /tests/caddyfile.bats, /tests/deployment_scoping.bats, /tests/deployment_flow.bats, /tests/frank_large_config.bats — Added migration coverage and corrected the existing ubatch harness assertion.
• /docs/index.md — Bundle index remains the root navigation entry for the updated pages.
• Boot-persistence audit (post-reboot, 2026-08-26 evening): after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (restart: unless-stopped does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): net.ipv4.ip_unprivileged_port_start was reset to 80 by two sources — a later-sorted sysctl drop-in (99-rootless-ports.conf) and a stale line in /etc/sysctl.conf, which systemd-sysctl applies after all of sysctl.d/ (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the /etc/sysctl.conf line was neutralized too).
• /AGENTS.md — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (WantedBy=default.target + enabled + linger, Restart=always, repo-provenance .container), verified before a service is declared deployed.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (caddy.service): joins dankclaw_default so container-name upstreams keep resolving, Pull=never (local-only image), Porkbun credentials via /home/dank/Containers/caddy/env (deploy writes it from .env), bare-IP-catch-all health check. The caddy service was removed from /dankclaw/compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — repo source of truth for ip_unprivileged_port_start=53; deploy_pihole installs it to /etc/sysctl.d/, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale /etc/sysctl.conf line (applied after all of sysctl.d), and applies sysctl --system (the boot code path).
• /scripts/deploy-remote.sh — deploy_caddy installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the default.target.wants symlink from the unit's [Install] section at every user-manager start — is-enabled reports generated for all healthy quadlets, and an explicit systemctl enable fails on generated units (learned live during the deploy). deploy_pihole enforces the sysctl prerequisites.
• /tests/caddy_quadlet.bats (new seam), /tests/caddyfile.bats, /tests/deployment_flow.bats, /tests/pihole.bats — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
• /docs/Infrastructure/Caddy.md, /docs/Infrastructure/Pi-hole.md, /docs/Dankclaw/Dankclaw Stack.md — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.