serverAppConfigs/serverAppConfigs-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Bundle Change Log	Chronological record of meaningful changes to the serverAppConfigs knowledge base.	changelog documentation maintenance	2026-09-14

Bundle Change Log

2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired

• The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so frank-small is retired and frank-medium took the :7770 endpoint.
• Runtime: charlie12345/ROCmFPX llama.cpp c49ebdb built for gfx1201 with the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via /etc/ld.so.conf.d/10-rocm-opencl.conf) is fully bypassed. Model: kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) + Q4_0 MTP draft head; mmproj unused (vision stays on frank-large).
• Parallelism decision, measured not argued: llama-bench with the correct multi-device syntax (-dev ROCm0/ROCm1 — comma form silently benches each card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse (no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline parallelism"); the data agrees on both axes. Draft head pinned to ROCm1 (card holding the last target layers); under tensor mode an unpinned draft aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp).
• Verified live before and after the systemd cutover: health ~18 s, MTP acceptance 60–69 %, cold 949-token prefill in 2036 ms, and prefix caching works with the draft head (turn 2 reprocessed 22 tokens, cache_n 945) — the model card's mtp-prompt-cache-fix.patch defect is already fixed in c49ebdb (common_prompt_checkpoint::data_spec).
• Deploy: deploy.sh dankserver --service frank-medium (new retire_root_llm_small() archived the CUDA-era unit to ~/.AGENTTRASH/20260915-frank-small-retire/host-units/); e2e gate 30/30 including 11-admin (the blankpad→192.168.0.112 path issue has cleared).
• Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo ~24.04 ROCm 6.3.3 packages, removed the radeon apt pin and the ld.so.conf hijack (staged in ~/.AGENTTRASH/20260915-rocm633-cleanup/); libhsa-runtime64 now resolves only to Ubuntu's ROCm 7.1 and rocm-smi 7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained).
• Client cutover: blankpad models.yml/config.yml renamed the model id to frank-medium (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw repointed dankserver-small routes to frank-medium.
• New: /serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md runbook; updated AGENTS.md, README, docs indexes, Dankserver System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and kingjones-q8agent marked retired/superseded.

2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw

• Added manualdousuario/marreta (Laravel / serversideup/php:8.4-fpm-nginx) as a rootless Quadlet pair: marreta + marreta-browser (Lightpanda), both joining gluetun's netns so all fetch egress exits the same ProtonVPN tunnel as SearXNG.
• Port-collision fix: the shared netns already has SearXNG on 8080, so marreta is relocated to NGINX_HTTP_PORT=8085 (verified against the serversideup source: the image does not bake conf.d/*.conf; the entrypoint envsubsts ${NGINX_HTTP_PORT} at start). Gluetun publishes 8082:8085 on marreta's behalf; Caddy dials 192.168.0.143:8082.
• Egress rationale: netns sharing, not PROXY_URL — the browser's CDP fetch passes no proxy, so only netns keeps the browser off the home IP.
• Guard chain: marreta.franksdank.cloud → Anubis :8923 → Authentik → post-auth :8017 @marreta → 192.168.0.143:8082. Access model is any authenticated user (no group gate). Requires a one-time ProxyProvider (external_host https://marreta.franksdank.cloud/, internal_host http://caddy:8017) on the embedded outpost.
• Secrets seeded to /home/dank/Containers/marreta/env (0600, dank-owned) from MARRETA_APP_KEY / MARRETA_ADMIN_EMAIL / MARRETA_ADMIN_PASSWORD.
• New test seam tests/marreta_quadlet.bats (17 contracts). deploy-plan.py: marreta joins the gluetun netns family. Doc: Marreta.
• Deployed live 2026-09-12 (rev 91e9410) after standing up passwordless SSH from blankpad to the fleet. Ran deploy.sh dankclaw --service gluetun --service marreta --service caddy (gluetun first: the new 8082:8085 publish needs a tunnel recreate, which also cycles SearXNG). Authentik ProxyProvider marreta registered (pk 13). Verified: unauth → Anubis challenge; post-auth :8017 → 200; container healthy; netns sharing proven; SearXNG unaffected.
• Gotcha fixed in-flight: the first seed used a bare-base64 APP_KEY, which 500s every Laravel app route (Unsupported cipher or incorrect key length). Re-keyed with the required base64: prefix before any encrypted data persisted. See Marreta § Secrets.
• Login-wall fixed 2026-09-12 (post-deploy). The homepage Marreta link parked on marreta.franksdank.cloud/if/flow/default-authentication-flow/ instead of the app. Root cause chain, in the order found:
  1. marreta-proxy had invalidation_flow = null, which crashed the Rust embedded outpost's serde (invalid type: null, expected a formatted UUID string) and took down every provider on the outpost. Set to default-provider-invalidation-flow.
  2. authentication_flow / authorization_flow were null; set to default-authentication-flow / default-provider-authorization-implicit-consent.
  3. The real blocker: the outpost's config API filters ProxyProvider.objects.filter(application__isnull=False). marreta-proxy had no linked Application, so the outpost never loaded it and the request fell through to raw flow serving on the marreta host — where the IdP session cookie (scoped to auth.franksdank.cloud) is invisible. Created Application(name="Marreta", slug="marreta", provider=marreta-proxy).
• After systemctl --user restart authentik-server authentik-outpost, the outpost loaded marreta-proxy and the OAuth dance completes: / → /outpost.goauthentik.io/start → auth…/o/authorize → callback → Marreta. Regression test: tests/e2e/specs/16-marreta.spec.ts. Providers remain live-only (no repo blueprint) — a provenance gap to close.

2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated

• Operator decision: *FrankBook-Pro is the sole arr deployment. The Dankserver compose copy (booted earlier the same day) was removed; /opt config roots trashed on the host; compose/services pruned.
• All six fleet containers (qB + five arrs) now share gluetun's network namespace — all egress exits the ProtonVPN tunnel by construction (AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live: residential 71.227.227.172 vs netns-wide 51.179.177.14.
• New test seam tests/frankbook_quadlet.bats: every arr+qB unit must join gluetun's netns and publish no port; gluetun must publish the whole fleet's port set. deploy-plan.py: gluetun changes carry the whole netns family.
• arr→qB download clients re-pointed 10.88.0.1:8080 (old bridge-gateway hack, dead inside the tunnel) → 127.0.0.1:8080; verified by real qB login from inside the netns. Caddy vhosts + homepage probes repoint to 192.168.0.112; homepage all green.
• Defects discovered: Prowlarr's four Application rows are GONE (empty table; sync broken until rebuilt with loopback baseUrls) and readarr's qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block parking may be stale now that egress is VPN — retest before enabling.
• E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now derive from the live services.json registry. AGENTS.md: deployments are complete only when unit tests AND Playwright E2E both pass.

2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented

• ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never booted — root-owned /opt/<svc>/config dirs crashed every app at logger init before any port bound) is live as of 2026-09-04 behind six own-auth TLS vhosts (<svc>.franksdank.cloud), all verified 200, homepage probes green. Documented the verified rootless identity model — container root (PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0 (/proc/<pid>/uid_map = 0 1000 1, 1 100000 65536), so data roots are 1000:1000 and userns_mode must never be added to compose services (podman-compose 1.5.0 runs services in pods; --userns + --pod is rejected). FrankBook-Pro differences noted: no gluetun egress gate on the Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user section and the rest in admin.

2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)

• Scope: every page under /serverAppConfigs-docs/ re-verified against live hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits only — no pages created, renamed, or deleted. Per the deploy planner, a docs-only change is a deploy no-op.
• Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
  • gluetun run on FrankBook-Pro (units installed to /home/core/.config/containers/systemd/ via the frankbook-pro deploy handler; PUID=0/PGID=0, data roots /Volumes/COMPLETE/<svc>-data:/config; qB pinned 4.6.7 sharing gluetun's netns with 8080 + 6881 published on the gluetun unit; readarr pinned amd64-1.0.980; no vhosts — direct IP:port, exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080 guarded), Calibre-Web (0.6.27-ls397 + universal-calibre mod; no standalone calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
• Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 → internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana, prometheus, loki, seerr) route post-auth to container-name upstreams; exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP, ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole basicauth edge → :8083).
• Gitea.md: data roots + secrets corrected to the live unit paths (/home/dank/Containers/gitea/{data,postgres,env}); UID gotcha corrected — the units carry no User=/UserNS override, so container uid 1000 is literally host uid 1000 (no subuid offset for these containers).
• Dedupe: Prowlarr.md/prowlarr.md collapsed into one canonical lowercase page via rename (git history retains both versions; index link unchanged).
• LLM docs: archive pages (DFlash DSpark Speculative Decoding, ling-3.0-flash-trial) reworded from current-tense to dated historical framing with pointers to the active runbook kingjones-strix-rocmfp4; LLM index.md + deployment.md fleet tables and health-gate text updated to the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771, health /v1/version — FLM has no /health route).
• Suite green after the sweep: ./tests/run.sh → 233 pass / 0 fail, including the docs-completeness seam that requires every page to stay index-linked.

2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs

• WireGuard credentials provisioned (second Proton config, server PA-12, endpoint 51.179.177.2:51820) → /home/core/Containers/gluetun/env (0600, VM-local); deploy.sh frankbook-pro --service gluetun --service qbittorrent brought both up; egress verified (Proton PA exit, ~19 MB/s on a live torrent). The .conf stays in ~/Downloads; the private key never enters the repo.
• gvproxy binds wildcard *:6881/udp (Go sets SO_REUSEADDR on TCP but not UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path. macOS application firewall still filters gvproxy inbound from non-local sources (stealth mode) — cross-host 8080 needs socketfilterfw allow.
• qB wiring: first-seed bug (hash under legacy WebUI\Password key, ignored by qB ≥ 4.2) fixed live (WebUI\Password_PBKDF2) and in scripts/deploy-remote.sh; DefaultSavePath repointed to /Volumes/COMPLETE/Finished Torrents — qB 5.2 API adds with a category do not inherit the category save path (no auto-TMM); download clients created in all four arrs (sonarr/radarr/lidarr via API, tests pass; readarr by direct readarr.db insert — its EOL validator rejects qB 5's 204 login as auth failure); Prowlarr Applications sync sonarr/radarr/ lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema). Legacy Prowlarr-* indexer rows (hostname prowlarr:9696, unresolvable on the Mac default network) deleted. E2E: Sintel fetched + downloaded through the tunnel at ~19 MB/s, completed into Finished Torrents.
• Docs: FrankBook-Pro rewritten to the active-tunnel reality.

2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config

• RDAP: 149.102.254.77 = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle hosting partner); Dankclaw's tunnel v6 2a07:b944::2:2 sits in CH-PROTONVPN-20160719; observed exit IP 149.22.88.19 = CDNEXT-SEA-CG. The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is no peer-registration step on any box of ours (full 65535-port scan: no SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770, 8443 - Proton's protocol-stealth port map, no admin surface).
• Mac gate corrected: provision a second Proton WG config (dashboard, named e.g. frankbook) - one config = one keypair; two hosts on Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values paste straight into the gluetun custom env. Earlier prepared keypair on dankclaw (~/frankbook-wg/) retired to ~/.AGENTTRASH/ (Proton generates the keypair per config; no BYO pubkey).
• FrankBook-Pro provisioning section rewritten.

2026-09-01 — Jellyfin authentication moved to Authentik LDAP

• /dankclaw/authentik/quadlet/authentik-outpost-ldap.container (4ad85fa, digest-pinned in f4ef2b8) — new authentik-outpost-ldap.service on Dankclaw serving LDAP :389/:636 for Jellyfin. Image ghcr.io/goauthentik/ldap pinned by amd64 single-platform digest: podman 5.7.0 refuses the tag because it resolves to a nested manifest list ("unexpectedly received a manifest list instead of a manifest for a single image") — resolve via podman manifest inspect | jq and pin.
• Authentik objects provisioned via ak shell ORM (live, idempotent get_or_create): LDAP provider Jellyfin LDAP, service account svc-jellyfin-ldap + bind token, groups jellyfin_users / jellyfin_admins, application jellyfin.
• /scripts/deploy-remote.sh (691da47) — authentik-outpost-ldap added to service_is_allowed (the deploy runner has its own allowlist beyond deploy-plan.py's).
• Two authentik-side rules proven by live bind tests (ldapsearch through the published port):
  1. Bind token intent must be verification — app_password tokens are rejected by the authorization flow's password stage with Invalid password.
  2. Search requires the search_full_directory object permission on the provider (granted via RBAC role svc-jellyfin-ldap-search); without it the outpost returns only the bind user, so memberOf filters match zero entries.
• Jellyfin side (host-only, no repo change): LDAP-Auth 23.0.0.0 installed from the GitHub releases zip (the repo.jellyfin.org stable manifest path 404s), config XML at config/plugins/configurations/LDAP-Auth.xml, username attribute cn (authentik 2026.x maps uid to an opaque hash), and — the big one — LdapSearchAttributes is comma-separated (Split(',') in the plugin source; semicolons silently produce "Found no users matching X"). End-to-end verified with a throwaway jellyfin_users member (AuthenticateByName 200, non-admin policy), then removed from both authentik and the Jellyfin DB directly (its admin-token cleanup 401'd — the admin password in the session archive was OCR-corrupted; verified offline against the stored PBKDF2-SHA512 hash that the real bootstrap value still authenticates, unchanged).
• Docs: Authentik gained the LDAP-outpost section (objects table + rules), Jellyfin gained the LDAP-Auth plugin section. Docs-only commit → deploy no-op.

2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored

• /scripts/frank-small.env, /scripts/serve-frank-small.sh, /scripts/frank-small.service (af6cd72 → 34ab6b4 → reverted in 4d5bea4) — attempted model swaps on FreeToken v0.1.2, both live-verified as loader failures and reverted the same day to the Huihui FP8 checkpoint:
  • ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors per-channel): ValueError: Missing MoE expert source layers: {'gate_up': ...} — the checkpoint's expert tensors do not follow the mlp.experts.{N}.gate_proj/up_proj/down_proj naming the MoE provider scans for.
  • lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 (compressed-tensors block-fp8 128×128, standard naming): experts resolve but _dequant_fp8_weight multiplies weight × weight_scale directly — block scales do not broadcast → RuntimeError: The size of tensor a (2048) must match ....
• Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no outage); service was back on Huihui within the hour and re-verified with a real completion (reasoning parses; 17*23 → 391, natural stop).
• /serverAppConfigs-docs/LLM/freetoken-frank-small.md — quant matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both compressed-tensors shapes documented with their exact errors.
• Deploy health-check trap (second occurrence): deploy.sh reported "healthy after approximately 6s" for BOTH failed loads — the health gate passes while the doomed worker is still starting. A model swap is only proven by the journal (Uvicorn startup) + a real completion.
• Staged for trash (awaiting purge approval): /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB) and /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8 (35 GB).

2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated

• Incident context: Dankserver hard-crashed and rebooted 2026-09-01; every bare compose-era media container (RomM trio, Jellyfin, Calibre-Web, Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule exists for. Recovery doubled as the Phase 1 cutover.
• Phase 1 (dankserver): six quadlet units shipped (dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/) + first-cutover deploy handlers (retire bare containers → rescue RomM's anonymous /romm volume → merge-move /opt/<svc> into /home/dank/Containers/<svc> → ordered start with health gates). All six verified: units active, is-enabled=generated (wants in the generator dir), ports 8085→8080, 8096, 8083, 3923 answering, jellyfin holds /dev/dri render nodes, entropy binds provably ro except copyparty's /w. MariaDB continuity proven via the real dank admin row (created 2026-08-31), not a fresh bootstrap. Stale compose-era container-*.service generate-systemd units retired to ~/.AGENTTRASH/20260901-stale-generate-systemd.
• Deployer findings fixed (each caught live, each with a new seam): v1 generator rejects Command= (→ Exec=), Device= (→ AddDevice=, whole-unit rejection is silent), and splits spaces in HealthCmd= (→ omitted for copyparty); migrate_data_root must merge into pre-seeded destinations and rename mapped-subuid roots (100998) with sudo mv (numeric-preserving, keep mode skips the chown — chowning mapped data breaks the in-container users); quadlet unit dir must be created on a host that never had quadlet; missing PublishPort = app answers inside the netns only (copyparty 3923); seeded env files must carry compose-provided vars only — the image-baked HOME=/root in a [Service] EnvironmentFile= kills the podman client itself.
• Phase 2 (dankclaw): data roots of gitea, authentik, hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime), units repointed, redeployed through deploy.sh --service; auth chain verified end-to-end (outpost ping → 302 to flow). deploy_hermes ordering bug fixed: generated units must be reloaded after unit files are copied, not before (stale ExecStart crash-looped the trio on the old path).
• Docs: RomM, Jellyfin, Calibre-Web, Copyparty updated to the quadlet reality; Storage Policy now documents the Containers layout
  • the two ownership regimes; Deployment Flow service-manager table corrected (dankserver media is no longer "bare containers, inventory only").

2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause

• Deployed frank-npu on Dankclaw: FLM v1.0.1 (/usr/bin/flm, dpkg fastflowlm) serving the whole local registry (36 tags) on port 52626; primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy crash-looped every 10 s: Open /dev/accel/accel0 failed (err=-19).
• Root cause (live-verified): the XDNA2 driver's SVA bind fails with ret -19 whenever the AMD IOMMU is off; the kernel cmdline carried iommu=off (a perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The same root cause explains the amdgpu ... no vmid pasid mapping supported kernel line — a consequence, not a GPU fault.
• Fix: iommu=pt in GRUB_CMDLINE_LINUX_DEFAULT + reboot (passthrough keeps the iGPU default domain identity-mapped — zero DMA-translation overhead for frank-large — while enabling IOMMU hardware + PASID, which SVA needs; PASID domains are paged regardless of the default-domain type). Post-boot: 30 IOMMU groups, zero SVA-bind errors since boot, unit enabled, NRestarts=0 since boot.
• Live-verified behavior: whole-registry serving with per-request routing by exact FLM tag (unknown tags silently serve llama3.2:1b); vision tower proved end-to-end (generated red PNG as image_url data-URI → correct color answer, 279 prompt tokens consumed); audio tower proven with a control pair (440 Hz tone via input_audio = 37 prompt tokens vs 15 for the identical text-only control). Health probe is /v1/version — FLM has no /health route.
• /scripts/frank-npu.env, /scripts/serve-frank-npu.sh, /scripts/frank-npu.service + deployer allowlist entries (c3ed3a6, cf0c541) — service files and deploy wiring; unit is boot-persistent (WantedBy=multi-user.target, enabled, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity — FLM requires unlimited memlock).
• Docs: new runbook frank-npu (linked from both indexes), AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard requirement. Client wiring: pi models.json registers the endpoint under the Dankclaw provider with the exact FLM registry tag as model id.

2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken

• Dankclaw / frank-large: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's pinned kingjones30/ROCmFPX runtime (built at /home/dank/GitPackages/kingjones-ROCmFPX/build, LLAMA_BUILD_WEBUI=OFF — npm/HF-bucket asset provisioning aborts the link at 98% otherwise). On-host 113.47 GiB file verified byte-identical to the HF LFS oid (no re-download). Kept our args (401408×2, ub 512, fa on, fit off); fa-on works UNPATCHED on this model — the QSA F32 assert was agentionai-specific. No spec decode (card ships none; MTP heads unloadable). Validated live: ~225 s load, health 200, reasoning completions, real 111k-token prefill at 213–232 t/s. Deployed via git flow; manual-process race during validation caught and fixed (unit was crash-looping on a bound port while its health check passed — check MainPID, not just /health).
• Dankserver / frank-small: swapped the model within FreeToken v0.1.2 to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8, same loader path and pool plan; 35 GB downloaded via hf, official checkpoint retained for rollback). Learned: FreeToken's chat API 422s without an explicit model field. ~40 tok/s verified post-deploy. Both .service Descriptions refreshed.
• Docs: kingjones-strix-rocmfp4 reactivated as the large runbook (runtime build + swap procedure + validation), agentionai-fast-imatrix marked RETIRED (rollback record), freetoken-frank-small model sections updated, LLM index + AGENTS fleet tables refreshed.

2026-08-30 (late) — download chain proven end to end; qB credentials persisted

• Cracked the arr↔Prowlarr feed contract: route is /{indexerId}/api (Prowlarr NewznabController; absolute /api/v1/indexer/{id}/newznab), auth = the Prowlarr global key. The previously hand-rolled /torznab/{id}/feed paths were Forms-auth catch-alls returning 302 → /login — which the arrs XML-parsed as "unexpected token '/'". A source-verified librarian answer settled it in one shot.
• Re-pointed all four arr indexer rows (Radarr→YTS id7, Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API; every PUT passed the app's live connection validator (202); RSS enabled.
• TV + music chains proven by live search: Sonarr returned 76 current releases via Prowlarr-Nyaa; Lidarr initially logged 0 active indexers (new rows default all-off) — enabling the search flags yielded 18 releases for YOASOBI – THE BOOK.
• First real grab: MoviesSearch → 3 YTS releases → The Dark Knight (2008) 2160p (7.5 GiB) → qBittorrent 100 % → imported (hasFile: true, 8,078,103,833 B at /movies/The Dark Knight (2008), hardlinked from the shared /mnt/entropy filesystem), queue empty.
• Prowlarr Application auto-sync found broken by version skew (Unable to cast object of type 'NzbDrone.Core.Thing…' against Radarr 6.3) — the manual rows are the integration; documented.
• qBittorrent permanent credentials installed: the WebUI temp password rotated on every boot (it had silently broken the arr clients once this session). qb5 removed the API password endpoint and the lscr image has no env support — wrote WebUI\Password_PBKDF2 (source-verified base64(salt):base64(PBKDF2-HMAC-SHA512/100k) form) into the conf with the container stopped; login survives podman restart (204). All arr download clients updated to it; value kept host-local in /home/dank/.qbpass (0600), never in git.
• Readarr parked with reason: abandoned upstream; its qB client cannot authenticate against qB 5 while three sibling apps pass with identical credentials. Indexer row healthy; books pipeline awaits a successor.
• Docs: rewrote Dankserver/prowlarr.md (live wiring + six verified gotchas incl. the joiner bug and the 302-masquerading-as-auth failure) and Dankserver/qbittorrent.md (persistent-credential recipe, Referer CSRF note).

2026-08-29 (late) — guarded-host sign-in fixed end to end

• Diagnosed both user symptoms: Invalid next URL = the flow being entered with an absolute cross-host next= (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had no auth A record (added via Porkbun API, auth A 71.227.227.172 TTL 300).
• Disproved meta-header routing live (full meta-header set still 404s; matching Host alone answers). The 2026.8 embedded outpost routes solely on the original Host vs each proxy provider's external_host.
• Provisioned proxy providers + applications for search, hermes, prometheus, romm, files (pks 2–6) via ORM, attached to the embedded outpost; internal_host = http://caddy:8017 everywhere.
• Wedge found + fixed live: providers created with null authorization_flow/invalidation_flow starve the outpost's config fetch (Failed to fetch outpost from API, retrying in 3 seconds forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
• /dankclaw/Caddyfile (29f9a86) — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; deleted the per-vhost redirect/flow design and stale comments.
• /dankclaw/anubis/policy.json (2a72efa) — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance; app paths remain challenged.
• Gotcha: the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile). Suite green (0 not-ok).
• /docs/Infrastructure/Authentik.md, /docs/Infrastructure/Anubis.md, /docs/Architecture/Network Architecture.md (ee1e038) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
• Verified by headless JS client: https://search.franksdank.cloud/ → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 state JWT. Credential submission is the user's step.

2026-09-14

• Creation: Hermes-WebUI — new hermes-webui compose service on Dankclaw (the Hermex server): runs hermes-webui's bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data with the gateway (identical rootless Podman uid mapping), publishes 127.0.0.1:8787 only, password-gated via HERMES_WEBUI_PASSWORD in the host .env.
• /dankclaw/compose.yaml, /dankclaw/.env.example — service definition and password placeholder; container-internal bind is 0.0.0.0 (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
• /tests/hermes_webui.bats — new seam: foreground entrypoint, loopback publish + auth, shared volume, /health healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
• Remote access pattern documented: iPhone Termius local forward 127.0.0.1:8787 -> 127.0.0.1:8787 over the existing router port forward to Dankclaw :22; Hermex points at http://localhost:8787. No Tailscale, no Cloudflare.

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts now proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
• /docs/Infrastructure/Caddy.md — Documented the Plex TLS-upstream requirement and the /web/ vs / distinction.
• /tests/caddyfile.bats — Added a test pinning the TLS upstream transport on both Plex vhosts.
• /docs/Dankclaw/Gitea.md — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
• /docs/Dankclaw/Dankclaw Stack.md — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
• /docs/Architecture/Deployment Flow.md — Replaced the split-proxy safety description with the canonical Caddy ownership model.
• /dankclaw/gitea/quadlet/gitea-caddy.container — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
• /scripts/deploy-plan.py — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
• /scripts/deploy.sh and /scripts/deploy-remote.sh — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
• /tests/caddyfile.bats, /tests/deployment_scoping.bats, /tests/deployment_flow.bats, /tests/frank_large_config.bats — Added migration coverage and corrected the existing ubatch harness assertion.
• /docs/index.md — Bundle index remains the root navigation entry for the updated pages.
• Boot-persistence audit (post-reboot, 2026-08-26 evening): after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (restart: unless-stopped does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): net.ipv4.ip_unprivileged_port_start was reset to 80 by two sources — a later-sorted sysctl drop-in (99-rootless-ports.conf) and a stale line in /etc/sysctl.conf, which systemd-sysctl applies after all of sysctl.d/ (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the /etc/sysctl.conf line was neutralized too).
• /AGENTS.md — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (WantedBy=default.target + enabled + linger, Restart=always, repo-provenance .container), verified before a service is declared deployed.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (caddy.service): joins dankclaw_default so container-name upstreams keep resolving, Pull=never (local-only image), Porkbun credentials via /home/dank/Containers/caddy/env (deploy writes it from .env), bare-IP-catch-all health check. The caddy service was removed from /dankclaw/compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — repo source of truth for ip_unprivileged_port_start=53; deploy_pihole installs it to /etc/sysctl.d/, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale /etc/sysctl.conf line (applied after all of sysctl.d), and applies sysctl --system (the boot code path).
• /scripts/deploy-remote.sh — deploy_caddy installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the default.target.wants symlink from the unit's [Install] section at every user-manager start — is-enabled reports generated for all healthy quadlets, and an explicit systemctl enable fails on generated units (learned live during the deploy). deploy_pihole enforces the sysctl prerequisites.
• /tests/caddy_quadlet.bats (new seam), /tests/caddyfile.bats, /tests/deployment_flow.bats, /tests/pihole.bats — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
• /docs/Infrastructure/Caddy.md, /docs/Infrastructure/Pi-hole.md, /docs/Dankclaw/Dankclaw Stack.md — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.

2026-09-01

• dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to /home/dank/Containers/<svc>, /opt/{romm,jellyfin,calibre-web,copyparty} retired, all four endpoints answering, legacy container-* units gone. Dankserver deploy-state for the six services records deployed revisions through 1a257c0.
• /scripts/deploy-remote.sh — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 @ByteArray WebUI password (plaintext is ignored by qB ≥ 4.2), sets HostHeaderValidation=false + ServerDomains=* for the socat forwarder path, and pre-creates the four categories with save_path=/Volumes/COMPLETE/Finished Torrents.
• frankbook-pro — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
• jellyfin GPU posture on dankserver: AddDevice=/dev/dri + GroupAdd=keep-groups in the unit (parity with pre-migration reality — no /dev/nvidia* in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
• /docs/Infrastructure/FrankBook-Pro.md (new), /docs/index.md — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
• /dankclaw/seerr/quadlet/seerr.container — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
• /dankclaw/Caddyfile, /dankclaw/pihole/addn-hosts — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider seerr-proxy (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
• /scripts/deploy-remote.sh — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
• /tests/seerr_quadlet.bats — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
• /serverAppConfigs-docs/Dankclaw/Seerr.md, /serverAppConfigs-docs/index.md — service doc + index entry.
• Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
• Authentik outpost claim cache fixed (root cause of the persistent /admin 403 despite homepage-admins membership): the outpost serves per-browser-session claims from authentik_providers_proxy_proxysession until row expiry — akadmin's live row predated the group add and carried ["authentik Admins"] only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in /serverAppConfigs-docs/Infrastructure/Authentik.md (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
• WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway 10.88.0.1:8080; Prowlarr's client created via its /downloadclient/schema template (hand-minimal body 400s on create-time test).
• /frankbook-pro/qbittorrent/quadlet/qbittorrent.container — pinned lscr.io/linuxserver/qbittorrent:4.6.7: latest had resolved to qB 5.x, whose 204 empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy 200 + "Ok.". Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login invalid credentials; rewriting the same line fresh (same password, 100k iterations) fixed it.
• /docs/Infrastructure/FrankBook-Pro.md — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into Finished Torrents, deleted with data).
• /scripts/deploy.sh + /scripts/deploy-plan.py + /tests/dankserver_quadlet.bats — committed earlier today (b206f47): dankserver quadlet standardization units/handlers/routing.

2026-09-04 — Sonarr moved to the admin side; internal-only by construction

• Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an admin-side service and must be locally reachable only. The homepage registry flipped its card from the user category to admin and repointed the link at http://192.168.0.112:8989 (LAN-direct). Users keep requesting TV through Seerr, which stays on the user side behind the guard chain.
• Exposure closure: the unguarded public vhost sonarr.franksdank.cloud (direct reverse_proxy 192.168.0.112:8989, no auth) and its post-auth @sonarr handler were retired from the Caddyfile; the Pi-hole addn-hosts A record for the name was dropped with it. No public Porkbun record ever existed (NXDOMAIN at the authoritative zone), so after this change there is no WAN path to Sonarr at all — manual DNS could previously reach the vhost, which served the zero-auth UI to anyone who resolved the name manually at the public edge; the vhost (added in e0bade6, 2026-09-04) is retired.