serverAppConfigs/serverAppConfigs-docs/log.md

---
type: reference
title: Bundle Change Log
description: Chronological record of meaningful changes to the serverAppConfigs knowledge base.
tags:
  - changelog
  - documentation
  - maintenance
timestamp: 2026-09-14
---

# Bundle Change Log

## 2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw
- Added `manualdousuario/marreta` (Laravel / `serversideup/php:8.4-fpm-nginx`)
  as a rootless Quadlet pair: `marreta` + `marreta-browser` (Lightpanda),
  both joining **gluetun's netns** so all fetch egress exits the same
  ProtonVPN tunnel as [[SearXNG]].
- **Port-collision fix:** the shared netns already has SearXNG on 8080, so
  marreta is relocated to `NGINX_HTTP_PORT=8085` (verified against the
  serversideup source: the image does not bake `conf.d/*.conf`; the
  entrypoint `envsubst`s `${NGINX_HTTP_PORT}` at start). Gluetun publishes
  `8082:8085` on marreta's behalf; Caddy dials `192.168.0.143:8082`.
- **Egress rationale:** netns sharing, not `PROXY_URL` — the browser's CDP
  fetch passes no proxy, so only netns keeps the browser off the home IP.
- **Guard chain:** `marreta.franksdank.cloud` → Anubis `:8923` → Authentik
  → post-auth `:8017` `@marreta` → `192.168.0.143:8082`. Access model is
  **any authenticated user** (no group gate). Requires a one-time
  `ProxyProvider` (external_host `https://marreta.franksdank.cloud/`,
  internal_host `http://caddy:8017`) on the embedded outpost.
- Secrets seeded to `/home/dank/Containers/marreta/env` (0600, dank-owned)
  from `MARRETA_APP_KEY` / `MARRETA_ADMIN_EMAIL` / `MARRETA_ADMIN_PASSWORD`.
- New test seam `tests/marreta_quadlet.bats` (17 contracts). deploy-plan.py:
  marreta joins the gluetun netns family. Doc: [[Marreta]].
- **Deployed live 2026-09-12 (rev `91e9410`)** after standing up passwordless
  SSH from blankpad to the fleet. Ran `deploy.sh dankclaw --service gluetun
  --service marreta --service caddy` (gluetun first: the new `8082:8085`
  publish needs a tunnel recreate, which also cycles SearXNG). Authentik
  `ProxyProvider` `marreta` registered (pk 13). Verified: unauth → Anubis
  challenge; post-auth `:8017` → 200; container healthy; netns sharing proven;
  SearXNG unaffected.
- **Gotcha fixed in-flight:** the first seed used a bare-base64 `APP_KEY`,
  which 500s every Laravel app route (`Unsupported cipher or incorrect key
  length`). Re-keyed with the required `base64:` prefix before any encrypted
  data persisted. See [[Marreta]] § Secrets.

## 2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated
- Operator decision: **FrankBook-Pro is the sole *arr deployment.** The
  Dankserver compose copy (booted earlier the same day) was removed;
  `/opt` config roots trashed on the host; compose/services pruned.
- All six fleet containers (qB + five arrs) now share gluetun's network
  namespace — **all egress exits the ProtonVPN tunnel by construction**
  (AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI
  ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live:
  residential `71.227.227.172` vs netns-wide `51.179.177.14`.
- New test seam `tests/frankbook_quadlet.bats`: every arr+qB unit must join
  gluetun's netns and publish no port; gluetun must publish the whole
  fleet's port set. deploy-plan.py: gluetun changes carry the whole netns
  family.
- arr→qB download clients re-pointed `10.88.0.1:8080` (old bridge-gateway
  hack, dead inside the tunnel) → `127.0.0.1:8080`; verified by real qB
  login from inside the netns. Caddy vhosts + homepage probes repoint to
  192.168.0.112; homepage all green.
- Defects discovered: Prowlarr's four Application rows are GONE (empty
  table; sync broken until rebuilt with loopback baseUrls) and readarr's
  qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block
  parking may be stale now that egress is VPN — retest before enabling.
- E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now
  derive from the live services.json registry. AGENTS.md: deployments are
  complete only when unit tests AND Playwright E2E both pass.

## 2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented
- ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never
  booted — root-owned `/opt/<svc>/config` dirs crashed every app at logger
  init before any port bound) is live as of 2026-09-04 behind six own-auth
  TLS vhosts (`<svc>.franksdank.cloud`), all verified 200, homepage probes
  green. Documented the verified rootless identity model — container root
  (PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0
  (`/proc/<pid>/uid_map` = `0 1000 1`, `1 100000 65536`), so data roots are
  `1000:1000` and **`userns_mode` must never be added to compose services**
  (podman-compose 1.5.0 runs services in pods; `--userns` + `--pod` is
  rejected). FrankBook-Pro differences noted: no gluetun egress gate on the
  Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user
  section and the rest in admin.
## 2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)
- Scope: every page under `/serverAppConfigs-docs/` re-verified against live
  hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits
  only — no pages created, renamed, or deleted. Per the deploy planner, a
  docs-only change is a deploy no-op.
- Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
  + gluetun run on **FrankBook-Pro** (units installed to
  `/home/core/.config/containers/systemd/` via the frankbook-pro deploy
  handler; `PUID=0/PGID=0`, data roots `/Volumes/COMPLETE/<svc>-data:/config`;
  qB pinned `4.6.7` sharing gluetun's netns with 8080 + 6881 published on the
  gluetun unit; readarr pinned `amd64-1.0.980`; no vhosts — direct IP:port,
  exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080
  guarded), Calibre-Web (`0.6.27-ls397` + universal-calibre mod; no standalone
  calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
- Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 →
  internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana,
  prometheus, loki, seerr) route post-auth to container-name upstreams;
  exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP,
  ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole
  basicauth edge → :8083).
- `Gitea.md`: data roots + secrets corrected to the live unit paths
  (`/home/dank/Containers/gitea/{data,postgres,env}`); UID gotcha corrected —
  the units carry no `User=`/UserNS override, so container uid 1000 is literally
  host uid 1000 (no subuid offset for these containers).
- Dedupe: `Prowlarr.md`/`prowlarr.md` collapsed into one canonical lowercase
  page via rename (git history retains both versions; index link unchanged).
- LLM docs: archive pages (`DFlash DSpark Speculative Decoding`,
  `ling-3.0-flash-trial`) reworded from current-tense to dated historical
  framing with pointers to the active runbook [[kingjones-strix-rocmfp4]];
  LLM `index.md` + `deployment.md` fleet tables and health-gate text updated to
  the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui
  abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771,
  health `/v1/version` — FLM has no `/health` route).
- Suite green after the sweep: `./tests/run.sh` → 233 pass / 0 fail, including
  the docs-completeness seam that requires every page to stay index-linked.
## 2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs
- WireGuard credentials provisioned (second Proton config, server `PA-12`,
  endpoint `51.179.177.2:51820`) → `/home/core/Containers/gluetun/env`
  (0600, VM-local); `deploy.sh frankbook-pro --service gluetun --service
  qbittorrent` brought both up; egress verified (Proton PA exit, ~19 MB/s
  on a live torrent). The `.conf` stays in `~/Downloads`; the private key
  never enters the repo.
- gvproxy binds wildcard `*:6881/udp` (Go sets `SO_REUSEADDR` on TCP but not
  UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and
  were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path.
  macOS application firewall still filters gvproxy inbound from non-local
  sources (stealth mode) — cross-host 8080 needs `socketfilterfw` allow.
- qB wiring: first-seed bug (hash under legacy `WebUI\Password` key, ignored
  by qB ≥ 4.2) fixed live (`WebUI\Password_PBKDF2`) and in
  `scripts/deploy-remote.sh`; `DefaultSavePath` repointed to
  `/Volumes/COMPLETE/Finished Torrents` — qB 5.2 API adds with a category
  do not inherit the category save path (no auto-TMM); download clients
  created in all four arrs (sonarr/radarr/lidarr via API, tests pass;
  readarr by direct `readarr.db` insert — its EOL validator rejects qB 5's
  `204` login as auth failure); Prowlarr Applications sync sonarr/radarr/
  lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema).
  Legacy `Prowlarr-*` indexer rows (hostname `prowlarr:9696`, unresolvable
  on the Mac default network) deleted. E2E: Sintel fetched + downloaded
  through the tunnel at ~19 MB/s, completed into `Finished Torrents`.
- Docs: [[FrankBook-Pro]] rewritten to the active-tunnel reality.


## 2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config
- RDAP: `149.102.254.77` = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle
  hosting partner); Dankclaw's tunnel v6 `2a07:b944::2:2` sits in
  CH-PROTONVPN-20160719; observed exit IP `149.22.88.19` = CDNEXT-SEA-CG.
  The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is
  no peer-registration step on any box of ours (full 65535-port scan: no
  SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770,
  8443 - Proton's protocol-stealth port map, no admin surface).
-   Mac gate corrected: provision a **second** Proton WG config (dashboard,
  named e.g. `frankbook`) - one config = one keypair; two hosts on
  Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values
  paste straight into the gluetun `custom` env. Earlier prepared keypair on
  dankclaw (`~/frankbook-wg/`) retired to `~/.AGENTTRASH/` (Proton generates
  the keypair per config; no BYO pubkey).
- [[FrankBook-Pro]] provisioning section rewritten.

## 2026-09-01 — Jellyfin authentication moved to Authentik LDAP
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` (`4ad85fa`,
  digest-pinned in `f4ef2b8`) — new `authentik-outpost-ldap.service` on
  Dankclaw serving LDAP :389/:636 for Jellyfin. Image
  `ghcr.io/goauthentik/ldap` pinned by **amd64 single-platform digest**:
  podman 5.7.0 refuses the tag because it resolves to a nested manifest
  list ("unexpectedly received a manifest list instead of a manifest for a
  single image") — resolve via `podman manifest inspect | jq` and pin.
- Authentik objects provisioned via `ak shell` ORM (live, idempotent
  get_or_create): LDAP provider `Jellyfin LDAP`, service account
  `svc-jellyfin-ldap` + bind token, groups `jellyfin_users` /
  `jellyfin_admins`, application `jellyfin`.
- `/scripts/deploy-remote.sh` (`691da47`) — `authentik-outpost-ldap` added
  to `service_is_allowed` (the deploy runner has its own allowlist beyond
  deploy-plan.py's).
- Two authentik-side rules proven by live bind tests (`ldapsearch` through
  the published port):
  1. Bind token **intent must be `verification`** — `app_password` tokens
     are rejected by the authorization flow's password stage with
     `Invalid password`.
  2. Search requires the **`search_full_directory`** object permission on
     the provider (granted via RBAC role `svc-jellyfin-ldap-search`);
     without it the outpost returns only the bind user, so `memberOf`
     filters match zero entries.
- Jellyfin side (host-only, no repo change): LDAP-Auth **23.0.0.0**
  installed from the GitHub releases zip (the `repo.jellyfin.org` stable
  manifest path 404s), config XML at
  `config/plugins/configurations/LDAP-Auth.xml`, username attribute `cn`
  (authentik 2026.x maps `uid` to an opaque hash), and — the big one —
  **`LdapSearchAttributes` is comma-separated** (`Split(',')` in the plugin
  source; semicolons silently produce "Found no users matching X").
  End-to-end verified with a throwaway `jellyfin_users` member
  (`AuthenticateByName` 200, non-admin policy), then removed from both
  authentik and the Jellyfin DB directly (its admin-token cleanup 401'd —
  the admin password in the session archive was OCR-corrupted; verified
  offline against the stored PBKDF2-SHA512 hash that the real bootstrap
  value still authenticates, unchanged).
- Docs: [[Authentik]] gained the LDAP-outpost section (objects table +
  rules), [[Jellyfin]] gained the LDAP-Auth plugin section. Docs-only
  commit → deploy no-op.

## 2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh`,
  `/scripts/frank-small.service` (`af6cd72` → `34ab6b4` → reverted
  in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both
  live-verified as loader failures and reverted the same day to the Huihui
  FP8 checkpoint:
  - `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel):
    `ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the
    checkpoint's expert tensors do not follow the
    `mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider
    scans for.
  - `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8`
    (compressed-tensors block-fp8 128×128, standard naming): experts
    resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale`
    directly — block scales do not broadcast →
    `RuntimeError: The size of tensor a (2048) must match ...`.
- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no
  outage); service was back on Huihui within the hour and re-verified with
  a real completion (reasoning parses; `17*23 → 391`, natural stop).
- `/serverAppConfigs-docs/LLM/freetoken-frank-small.md` — quant
  matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both
  compressed-tensors shapes documented with their exact errors.
- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported
  "healthy after approximately 6s" for BOTH failed loads — the health gate
  passes while the doomed worker is still starting. A model swap is only
  proven by the journal (Uvicorn startup) + a real completion.
- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8`
  (37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8`
  (35 GB).

## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated
- **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every
  bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,
  Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule
  exists for. Recovery doubled as the Phase 1 cutover.
- **Phase 1 (dankserver)**: six quadlet units shipped
  (`dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/`) + first-cutover
  deploy handlers (retire bare containers → rescue RomM's anonymous `/romm`
  volume → merge-move `/opt/<svc>` into `/home/dank/Containers/<svc>` →
  ordered start with health gates). All six verified: units `active`,
  `is-enabled=generated` (wants in the generator dir), ports 8085→8080,
  8096, 8083, 3923 answering, jellyfin holds `/dev/dri` render nodes,
  entropy binds provably ro except copyparty's `/w`. MariaDB continuity
  proven via the real `dank` admin row (created 2026-08-31), not a fresh
  bootstrap. Stale compose-era `container-*.service` generate-systemd units
  retired to `~/.AGENTTRASH/20260901-stale-generate-systemd`.
- **Deployer findings fixed** (each caught live, each with a new seam):
  v1 generator rejects `Command=` (→ `Exec=`), `Device=` (→ `AddDevice=`,
  whole-unit rejection is silent), and splits spaces in `HealthCmd=`
  (→ omitted for copyparty); `migrate_data_root` must merge into pre-seeded
  destinations and rename mapped-subuid roots (100998) with `sudo mv`
  (numeric-preserving, `keep` mode skips the chown — chowning mapped data
  breaks the in-container users); quadlet unit dir must be created on a host
  that never had quadlet; missing `PublishPort` = app answers inside the
  netns only (copyparty 3923); seeded env files must carry **compose-provided
  vars only** — the image-baked `HOME=/root` in a `[Service]
  EnvironmentFile=` kills the podman client itself.
- **Phase 2 (dankclaw)**: data roots of gitea, authentik, hermes trios moved
  to `/home/dank/Containers/<svc>/` while running (rename is inode-bound —
  zero downtime), units repointed, redeployed through `deploy.sh --service`;
  auth chain verified end-to-end (outpost ping → 302 to flow).
  `deploy_hermes` ordering bug fixed: generated units must be reloaded
  **after** unit files are copied, not before (stale ExecStart crash-looped
  the trio on the old path).
- **Docs**: [[RomM]], [[Jellyfin]], [[Calibre-Web]], [[Copyparty]] updated to
  the quadlet reality; [[Storage Policy]] now documents the Containers layout
  + the two ownership regimes; [[Deployment Flow]] service-manager table
  corrected (dankserver media is no longer "bare containers, inventory only").

## 2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause
- **Deployed** `frank-npu` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`,
  dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626;
  primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy
  crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`.
- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19
  whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a
  perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU
  groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The
  same root cause explains the `amdgpu ... no vmid pasid mapping supported`
  kernel line — a consequence, not a GPU fault.
- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps
  the iGPU default domain identity-mapped — zero DMA-translation overhead for
  frank-large — while enabling IOMMU hardware + PASID, which SVA needs;
  PASID domains are paged regardless of the default-domain type). Post-boot:
  30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`,
  `NRestarts=0` since boot.
- **Live-verified behavior**: whole-registry serving with per-request routing by
  exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved
  end-to-end (generated red PNG as `image_url` data-URI → correct color answer,
  279 prompt tokens consumed); audio tower proven with a control pair (440 Hz
  tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only
  control). Health probe is `/v1/version` — FLM has no `/health` route.
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`,
  `/scripts/frank-npu.service` + deployer allowlist entries
  (`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent
  (`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`,
  `LimitMEMLOCK=infinity` — FLM requires unlimited memlock).
- **Docs**: new runbook [[frank-npu]] (linked from both indexes),
  AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard
  requirement. Client wiring: pi `models.json` registers the endpoint under the
  Dankclaw provider with the exact FLM registry tag as model id.

## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
- **Dankclaw / frank-large**: swapped to
  `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's
  pinned `kingjones30/ROCmFPX` runtime (built at
  `/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF`
  — npm/HF-bucket asset provisioning aborts the link at 98% otherwise).
  On-host 113.47 GiB file verified byte-identical to the HF LFS oid
  (no re-download). Kept our args (401408×2, ub 512, fa on, fit off);
  fa-on works UNPATCHED on this model — the QSA F32 assert was
  agentionai-specific. No spec decode (card ships none; MTP heads
  unloadable). Validated live: ~225 s load, health 200, reasoning
  completions, real 111k-token prefill at 213–232 t/s. Deployed via
  git flow; manual-process race during validation caught and fixed
  (unit was crash-looping on a bound port while its health check
  passed — check MainPID, not just /health).
- **Dankserver / frank-small**: swapped the model within
  FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
 (native block-fp8, same loader path and pool plan; 35 GB downloaded
  via `hf`, official checkpoint retained for rollback). Learned:
  FreeToken's chat API 422s without an explicit `model` field. ~40
  tok/s verified post-deploy. Both `.service` Descriptions refreshed.
- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook
  (runtime build + swap procedure + validation), [[agentionai-fast-imatrix]]
  marked RETIRED (rollback record), [[freetoken-frank-small]]
  model sections updated, LLM index + AGENTS fleet tables refreshed.

## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted
- **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api`
  (Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),
  auth = the Prowlarr **global** key. The previously hand-rolled
  `/torznab/{id}/feed` paths were Forms-auth catch-alls returning
  `302 → /login` — which the arrs XML-parsed as "unexpected token '/'".
  A source-verified librarian answer settled it in one shot.
- **Re-pointed all four arr indexer rows** (Radarr→YTS id7,
  Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API;
  every PUT passed the app's live connection validator (202); RSS enabled.
- **TV + music chains proven** by live search: Sonarr returned 76 current
  releases via Prowlarr-Nyaa; Lidarr initially logged `0 active indexers`
  (new rows default all-off) — enabling the search flags yielded 18
  releases for YOASOBI – THE BOOK.
- **First real grab**: `MoviesSearch` → 3 YTS releases → *The Dark Knight
  (2008) 2160p* (7.5 GiB) → qBittorrent 100 % → **imported**
  (`hasFile: true`, 8,078,103,833 B at `/movies/The Dark Knight (2008)`,
  hardlinked from the shared `/mnt/entropy` filesystem), queue empty.
- **Prowlarr Application auto-sync found broken by version skew**
  (`Unable to cast object of type 'NzbDrone.Core.Thing…'` against Radarr
  6.3) — the manual rows are the integration; documented.
- **qBittorrent permanent credentials installed**: the WebUI temp password
  rotated on every boot (it had silently broken the arr clients once this
  session). qb5 removed the API password endpoint and the lscr image has
  no env support — wrote `WebUI\Password_PBKDF2` (source-verified
  `base64(salt):base64(PBKDF2-HMAC-SHA512/100k)` form) into the conf with
  the container stopped; login survives `podman restart` (204). All arr
  download clients updated to it; value kept host-local in
  `/home/dank/.qbpass` (0600), never in git.
- **Readarr parked with reason**: abandoned upstream; its qB client cannot
  authenticate against qB 5 while three sibling apps pass with identical
  credentials. Indexer row healthy; books pipeline awaits a successor.
- **Docs**: rewrote `Dankserver/prowlarr.md` (live wiring + six verified
  gotchas incl. the joiner bug and the 302-masquerading-as-auth failure)
  and `Dankserver/qbittorrent.md` (persistent-credential recipe, Referer
  CSRF note).

## 2026-08-29 (late) — guarded-host sign-in fixed end to end
- **Diagnosed** both user symptoms: `Invalid next URL` = the flow being entered with an absolute cross-host `next=` (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had **no `auth` A record** (added via Porkbun API, `auth A 71.227.227.172 TTL 300`).
- **Disproved meta-header routing** live (full meta-header set still 404s; matching `Host` alone answers). The 2026.8 embedded outpost routes solely on the original `Host` vs each proxy provider's `external_host`.
- **Provisioned** proxy providers + applications for `search`, `hermes`, `prometheus`, `romm`, `files` (pks 2–6) via ORM, attached to the embedded outpost; `internal_host = http://caddy:8017` everywhere.
- **Wedge found + fixed live:** providers created with null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch (`Failed to fetch outpost from API, retrying in 3 seconds` forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
- `/dankclaw/Caddyfile` (`29f9a86`) — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; deleted the per-vhost redirect/flow design and stale comments.
- `/dankclaw/anubis/policy.json` (`2a72efa`) — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance; app paths remain challenged.
- **Gotcha:** the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). `--service anubis` deploy restart is mandatory after policy edits.
- `/tests/authentik.bats` — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile). Suite green (`0` not-ok).
- `/docs/Infrastructure/Authentik.md`, `/docs/Infrastructure/Anubis.md`, `/docs/Architecture/Network Architecture.md` (`ee1e038`) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
- **Verified** by headless JS client: `https://search.franksdank.cloud/` → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 `state` JWT. Credential submission is the user's step.

## 2026-09-14
- **Creation**: [[Hermes-WebUI]] — new `hermes-webui` compose service on Dankclaw (the Hermex server): runs `hermes-webui`'s `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data` with the gateway (identical rootless Podman uid mapping), publishes `127.0.0.1:8787` only, password-gated via `HERMES_WEBUI_PASSWORD` in the host `.env`.
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — service definition and password placeholder; container-internal bind is `0.0.0.0` (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
- `/tests/hermes_webui.bats` — new seam: foreground entrypoint, loopback publish + auth, shared volume, `/health` healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
- Remote access pattern documented: iPhone Termius local forward `127.0.0.1:8787 -> 127.0.0.1:8787` over the existing router port forward to Dankclaw `:22`; Hermex points at `http://localhost:8787`. No Tailscale, no Cloudflare.

## 2026-08-26
- `/dankclaw/Caddyfile` — Both Plex vhosts now proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
- `/docs/Infrastructure/Caddy.md` — Documented the Plex TLS-upstream requirement and the `/web/` vs `/` distinction.
- `/tests/caddyfile.bats` — Added a test pinning the TLS upstream transport on both Plex vhosts.
- `/docs/Dankclaw/Gitea.md` — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
- `/docs/Dankclaw/Dankclaw Stack.md` — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
- `/docs/Architecture/Deployment Flow.md` — Replaced the split-proxy safety description with the canonical Caddy ownership model.
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
- `/scripts/deploy-plan.py` — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start — `is-enabled` reports `generated` for all healthy quadlets, and an explicit `systemctl enable` fails on generated units (learned live during the deploy). `deploy_pihole` enforces the sysctl prerequisites.
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.

## 2026-09-01
- dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to `/home/dank/Containers/<svc>`, `/opt/{romm,jellyfin,calibre-web,copyparty}` retired, all four endpoints answering, legacy `container-*` units gone. Dankserver deploy-state for the six services records deployed revisions through `1a257c0`.
- `/scripts/deploy-remote.sh` — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 `@ByteArray` WebUI password (plaintext is ignored by qB ≥ 4.2), sets `HostHeaderValidation=false` + `ServerDomains=*` for the socat forwarder path, and pre-creates the four categories with `save_path=/Volumes/COMPLETE/Finished Torrents`.
- `frankbook-pro` — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
- jellyfin GPU posture on dankserver: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` in the unit (parity with pre-migration reality — no `/dev/nvidia*` in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
- `/docs/Infrastructure/FrankBook-Pro.md` (new), `/docs/index.md` — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider `seerr-proxy` (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
- `/scripts/deploy-remote.sh` — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
- `/tests/seerr_quadlet.bats` — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
- `/serverAppConfigs-docs/Dankclaw/Seerr.md`, `/serverAppConfigs-docs/index.md` — service doc + index entry.
- Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
- Authentik outpost claim cache fixed (root cause of the persistent `/admin` 403 despite homepage-admins membership): the outpost serves per-browser-session claims from `authentik_providers_proxy_proxysession` until row expiry — akadmin's live row predated the group add and carried `["authentik Admins"]` only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in `/serverAppConfigs-docs/Infrastructure/Authentik.md` (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
- WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway `10.88.0.1:8080`; Prowlarr's client created via its `/downloadclient/schema` template (hand-minimal body 400s on create-time test).
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — pinned `lscr.io/linuxserver/qbittorrent:4.6.7`: `latest` had resolved to qB 5.x, whose `204` empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy `200` + `"Ok."`. Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login `invalid credentials`; rewriting the same line fresh (same password, 100k iterations) fixed it.
- `/docs/Infrastructure/FrankBook-Pro.md` — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into `Finished Torrents`, deleted with data).
- `/scripts/deploy.sh` + `/scripts/deploy-plan.py` + `/tests/dankserver_quadlet.bats` — committed earlier today (`b206f47`): dankserver quadlet standardization units/handlers/routing.

## 2026-09-04 — Sonarr moved to the admin side; internal-only by construction
- Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an
  **admin-side** service and must be locally reachable only. The homepage
  registry flipped its card from the user category to **admin** and repointed
  the link at `http://192.168.0.112:8989` (LAN-direct). Users keep requesting
  TV through Seerr, which stays on the user side behind the guard chain.
- Exposure closure: the unguarded public vhost `sonarr.franksdank.cloud`
  (direct `reverse_proxy 192.168.0.112:8989`, no auth) and its post-auth
  `@sonarr` handler were retired from the Caddyfile; the Pi-hole addn-hosts A
  record for the name was dropped with it. No public Porkbun record ever
  existed (NXDOMAIN at the authoritative zone), so after this change there is
  no WAN path to Sonarr at all — manual DNS could previously reach the vhost,
  which served the zero-auth UI to anyone who resolved the name manually at
  the public edge; the vhost (added in `e0bade6`, 2026-09-04) is retired.