serverAppConfigs/serverAppConfigs-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Bundle Change Log	Chronological record of meaningful changes to the serverAppConfigs knowledge base.	changelog documentation maintenance	2026-09-14

Bundle Change Log

2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated

• Incident context: Dankserver hard-crashed and rebooted 2026-09-01; every bare compose-era media container (RomM trio, Jellyfin, Calibre-Web, Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule exists for. Recovery doubled as the Phase 1 cutover.
• Phase 1 (dankserver): six quadlet units shipped (dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/) + first-cutover deploy handlers (retire bare containers → rescue RomM's anonymous /romm volume → merge-move /opt/<svc> into /home/dank/Containers/<svc> → ordered start with health gates). All six verified: units active, is-enabled=generated (wants in the generator dir), ports 8085→8080, 8096, 8083, 3923 answering, jellyfin holds /dev/dri render nodes, entropy binds provably ro except copyparty's /w. MariaDB continuity proven via the real dank admin row (created 2026-08-31), not a fresh bootstrap. Stale compose-era container-*.service generate-systemd units retired to ~/.AGENTTRASH/20260901-stale-generate-systemd.
• Deployer findings fixed (each caught live, each with a new seam): v1 generator rejects Command= (→ Exec=), Device= (→ AddDevice=, whole-unit rejection is silent), and splits spaces in HealthCmd= (→ omitted for copyparty); migrate_data_root must merge into pre-seeded destinations and rename mapped-subuid roots (100998) with sudo mv (numeric-preserving, keep mode skips the chown — chowning mapped data breaks the in-container users); quadlet unit dir must be created on a host that never had quadlet; missing PublishPort = app answers inside the netns only (copyparty 3923); seeded env files must carry compose-provided vars only — the image-baked HOME=/root in a [Service] EnvironmentFile= kills the podman client itself.
• Phase 2 (dankclaw): data roots of gitea, authentik, hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime), units repointed, redeployed through deploy.sh --service; auth chain verified end-to-end (outpost ping → 302 to flow). deploy_hermes ordering bug fixed: generated units must be reloaded after unit files are copied, not before (stale ExecStart crash-looped the trio on the old path).
• Docs: RomM, Jellyfin, Calibre-Web, Copyparty updated to the quadlet reality; Storage Policy now documents the Containers layout
  • the two ownership regimes; Deployment Flow service-manager table corrected (dankserver media is no longer "bare containers, inventory only").

2026-08-30 (night II) — frank-default-sightsound deployed; NPU SVA/IOMMU root cause

• Deployed frank-default-sightsound on Dankclaw: FLM v1.0.1 (/usr/bin/flm, dpkg fastflowlm) serving the whole local registry (36 tags) on port 52626; primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy crash-looped every 10 s: Open /dev/accel/accel0 failed (err=-19).
• Root cause (live-verified): the XDNA2 driver's SVA bind fails with ret -19 whenever the AMD IOMMU is off; the kernel cmdline carried iommu=off (a perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The same root cause explains the amdgpu ... no vmid pasid mapping supported kernel line — a consequence, not a GPU fault.
• Fix: iommu=pt in GRUB_CMDLINE_LINUX_DEFAULT + reboot (passthrough keeps the iGPU default domain identity-mapped — zero DMA-translation overhead for frank-default-large — while enabling IOMMU hardware + PASID, which SVA needs; PASID domains are paged regardless of the default-domain type). Post-boot: 30 IOMMU groups, zero SVA-bind errors since boot, unit enabled, NRestarts=0 since boot.
• Live-verified behavior: whole-registry serving with per-request routing by exact FLM tag (unknown tags silently serve llama3.2:1b); vision tower proved end-to-end (generated red PNG as image_url data-URI → correct color answer, 279 prompt tokens consumed); audio tower proven with a control pair (440 Hz tone via input_audio = 37 prompt tokens vs 15 for the identical text-only control). Health probe is /v1/version — FLM has no /health route.
• /scripts/frank-default-sightsound.env, /scripts/serve-frank-default-sightsound.sh, /scripts/frank-default-sightsound.service + deployer allowlist entries (c3ed3a6, cf0c541) — service files and deploy wiring; unit is boot-persistent (WantedBy=multi-user.target, enabled, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity — FLM requires unlimited memlock).
• Docs: new runbook frank-default-sightsound (linked from both indexes), AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard requirement. Client wiring: pi models.json registers the endpoint under the Dankclaw provider with the exact FLM registry tag as model id.

2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken

• Dankclaw / frank-default-large: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's pinned kingjones30/ROCmFPX runtime (built at /home/dank/GitPackages/kingjones-ROCmFPX/build, LLAMA_BUILD_WEBUI=OFF — npm/HF-bucket asset provisioning aborts the link at 98% otherwise). On-host 113.47 GiB file verified byte-identical to the HF LFS oid (no re-download). Kept our args (401408×2, ub 512, fa on, fit off); fa-on works UNPATCHED on this model — the QSA F32 assert was agentionai-specific. No spec decode (card ships none; MTP heads unloadable). Validated live: ~225 s load, health 200, reasoning completions, real 111k-token prefill at 213–232 t/s. Deployed via git flow; manual-process race during validation caught and fixed (unit was crash-looping on a bound port while its health check passed — check MainPID, not just /health).
• Dankserver / frank-default-small: swapped the model within FreeToken v0.1.2 to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8, same loader path and pool plan; 35 GB downloaded via hf, official checkpoint retained for rollback). Learned: FreeToken's chat API 422s without an explicit model field. ~40 tok/s verified post-deploy. Both .service Descriptions refreshed.
• Docs: kingjones-strix-rocmfp4 reactivated as the large runbook (runtime build + swap procedure + validation), agentionai-fast-imatrix marked RETIRED (rollback record), freetoken-frank-default-small model sections updated, LLM index + AGENTS fleet tables refreshed.

2026-08-30 (late) — download chain proven end to end; qB credentials persisted

• Cracked the arr↔Prowlarr feed contract: route is /{indexerId}/api (Prowlarr NewznabController; absolute /api/v1/indexer/{id}/newznab), auth = the Prowlarr global key. The previously hand-rolled /torznab/{id}/feed paths were Forms-auth catch-alls returning 302 → /login — which the arrs XML-parsed as "unexpected token '/'". A source-verified librarian answer settled it in one shot.
• Re-pointed all four arr indexer rows (Radarr→YTS id7, Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API; every PUT passed the app's live connection validator (202); RSS enabled.
• TV + music chains proven by live search: Sonarr returned 76 current releases via Prowlarr-Nyaa; Lidarr initially logged 0 active indexers (new rows default all-off) — enabling the search flags yielded 18 releases for YOASOBI – THE BOOK.
• First real grab: MoviesSearch → 3 YTS releases → The Dark Knight (2008) 2160p (7.5 GiB) → qBittorrent 100 % → imported (hasFile: true, 8,078,103,833 B at /movies/The Dark Knight (2008), hardlinked from the shared /mnt/entropy filesystem), queue empty.
• Prowlarr Application auto-sync found broken by version skew (Unable to cast object of type 'NzbDrone.Core.Thing…' against Radarr 6.3) — the manual rows are the integration; documented.
• qBittorrent permanent credentials installed: the WebUI temp password rotated on every boot (it had silently broken the arr clients once this session). qb5 removed the API password endpoint and the lscr image has no env support — wrote WebUI\Password_PBKDF2 (source-verified base64(salt):base64(PBKDF2-HMAC-SHA512/100k) form) into the conf with the container stopped; login survives podman restart (204). All arr download clients updated to it; value kept host-local in /home/dank/.qbpass (0600), never in git.
• Readarr parked with reason: abandoned upstream; its qB client cannot authenticate against qB 5 while three sibling apps pass with identical credentials. Indexer row healthy; books pipeline awaits a successor.
• Docs: rewrote Dankserver/prowlarr.md (live wiring + six verified gotchas incl. the joiner bug and the 302-masquerading-as-auth failure) and Dankserver/qbittorrent.md (persistent-credential recipe, Referer CSRF note).

2026-08-29 (late) — guarded-host sign-in fixed end to end

• Diagnosed both user symptoms: Invalid next URL = the flow being entered with an absolute cross-host next= (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had no auth A record (added via Porkbun API, auth A 71.227.227.172 TTL 300).
• Disproved meta-header routing live (full meta-header set still 404s; matching Host alone answers). The 2026.8 embedded outpost routes solely on the original Host vs each proxy provider's external_host.
• Provisioned proxy providers + applications for search, hermes, prometheus, romm, files (pks 2–6) via ORM, attached to the embedded outpost; internal_host = http://caddy:8017 everywhere.
• Wedge found + fixed live: providers created with null authorization_flow/invalidation_flow starve the outpost's config fetch (Failed to fetch outpost from API, retrying in 3 seconds forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
• /dankclaw/Caddyfile (29f9a86) — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; deleted the per-vhost redirect/flow design and stale comments.
• /dankclaw/anubis/policy.json (2a72efa) — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance; app paths remain challenged.
• Gotcha: the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile). Suite green (0 not-ok).
• /docs/Infrastructure/Authentik.md, /docs/Infrastructure/Anubis.md, /docs/Architecture/Network Architecture.md (ee1e038) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
• Verified by headless JS client: https://search.franksdank.cloud/ → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 state JWT. Credential submission is the user's step.

2026-09-14

• Creation: Hermes-WebUI — new hermes-webui compose service on Dankclaw (the Hermex server): runs hermes-webui's bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data with the gateway (identical rootless Podman uid mapping), publishes 127.0.0.1:8787 only, password-gated via HERMES_WEBUI_PASSWORD in the host .env.
• /dankclaw/compose.yaml, /dankclaw/.env.example — service definition and password placeholder; container-internal bind is 0.0.0.0 (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
• /tests/hermes_webui.bats — new seam: foreground entrypoint, loopback publish + auth, shared volume, /health healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
• Remote access pattern documented: iPhone Termius local forward 127.0.0.1:8787 -> 127.0.0.1:8787 over the existing router port forward to Dankclaw :22; Hermex points at http://localhost:8787. No Tailscale, no Cloudflare.

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts now proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
• /docs/Infrastructure/Caddy.md — Documented the Plex TLS-upstream requirement and the /web/ vs / distinction.
• /tests/caddyfile.bats — Added a test pinning the TLS upstream transport on both Plex vhosts.
• /docs/Dankclaw/Gitea.md — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
• /docs/Dankclaw/Dankclaw Stack.md — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
• /docs/Architecture/Deployment Flow.md — Replaced the split-proxy safety description with the canonical Caddy ownership model.
• /dankclaw/gitea/quadlet/gitea-caddy.container — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
• /scripts/deploy-plan.py — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
• /scripts/deploy.sh and /scripts/deploy-remote.sh — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
• /tests/caddyfile.bats, /tests/deployment_scoping.bats, /tests/deployment_flow.bats, /tests/frank_large_config.bats — Added migration coverage and corrected the existing ubatch harness assertion.
• /docs/index.md — Bundle index remains the root navigation entry for the updated pages.
• Boot-persistence audit (post-reboot, 2026-08-26 evening): after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (restart: unless-stopped does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): net.ipv4.ip_unprivileged_port_start was reset to 80 by two sources — a later-sorted sysctl drop-in (99-rootless-ports.conf) and a stale line in /etc/sysctl.conf, which systemd-sysctl applies after all of sysctl.d/ (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the /etc/sysctl.conf line was neutralized too).
• /AGENTS.md — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (WantedBy=default.target + enabled + linger, Restart=always, repo-provenance .container), verified before a service is declared deployed.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (caddy.service): joins dankclaw_default so container-name upstreams keep resolving, Pull=never (local-only image), Porkbun credentials via /home/dank/Containers/caddy/env (deploy writes it from .env), bare-IP-catch-all health check. The caddy service was removed from /dankclaw/compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — repo source of truth for ip_unprivileged_port_start=53; deploy_pihole installs it to /etc/sysctl.d/, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale /etc/sysctl.conf line (applied after all of sysctl.d), and applies sysctl --system (the boot code path).
• /scripts/deploy-remote.sh — deploy_caddy installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the default.target.wants symlink from the unit's [Install] section at every user-manager start — is-enabled reports generated for all healthy quadlets, and an explicit systemctl enable fails on generated units (learned live during the deploy). deploy_pihole enforces the sysctl prerequisites.
• /tests/caddy_quadlet.bats (new seam), /tests/caddyfile.bats, /tests/deployment_flow.bats, /tests/pihole.bats — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
• /docs/Infrastructure/Caddy.md, /docs/Infrastructure/Pi-hole.md, /docs/Dankclaw/Dankclaw Stack.md — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.