Frank
32b4016e55
Flip the homepage Sonarr card from the user category to admin and repoint its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A record with it. No public Porkbun record ever existed (NXDOMAIN at the zone), so after this there is no WAN path to Sonarr: users request TV via Seerr (guarded), admins reach Sonarr's own UI from the LAN only. Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
407 lines
36 KiB
Markdown
407 lines
36 KiB
Markdown
---
|
||
type: reference
|
||
title: Bundle Change Log
|
||
description: Chronological record of meaningful changes to the serverAppConfigs knowledge base.
|
||
tags:
|
||
- changelog
|
||
- documentation
|
||
- maintenance
|
||
timestamp: 2026-09-14
|
||
---
|
||
|
||
# Bundle Change Log
|
||
|
||
## 2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated
|
||
- Operator decision: **FrankBook-Pro is the sole *arr deployment.** The
|
||
Dankserver compose copy (booted earlier the same day) was removed;
|
||
`/opt` config roots trashed on the host; compose/services pruned.
|
||
- All six fleet containers (qB + five arrs) now share gluetun's network
|
||
namespace — **all egress exits the ProtonVPN tunnel by construction**
|
||
(AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI
|
||
ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live:
|
||
residential `71.227.227.172` vs netns-wide `51.179.177.14`.
|
||
- New test seam `tests/frankbook_quadlet.bats`: every arr+qB unit must join
|
||
gluetun's netns and publish no port; gluetun must publish the whole
|
||
fleet's port set. deploy-plan.py: gluetun changes carry the whole netns
|
||
family.
|
||
- arr→qB download clients re-pointed `10.88.0.1:8080` (old bridge-gateway
|
||
hack, dead inside the tunnel) → `127.0.0.1:8080`; verified by real qB
|
||
login from inside the netns. Caddy vhosts + homepage probes repoint to
|
||
192.168.0.112; homepage all green.
|
||
- Defects discovered: Prowlarr's four Application rows are GONE (empty
|
||
table; sync broken until rebuilt with loopback baseUrls) and readarr's
|
||
qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block
|
||
parking may be stale now that egress is VPN — retest before enabling.
|
||
- E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now
|
||
derive from the live services.json registry. AGENTS.md: deployments are
|
||
complete only when unit tests AND Playwright E2E both pass.
|
||
|
||
## 2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented
|
||
- ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never
|
||
booted — root-owned `/opt/<svc>/config` dirs crashed every app at logger
|
||
init before any port bound) is live as of 2026-09-04 behind six own-auth
|
||
TLS vhosts (`<svc>.franksdank.cloud`), all verified 200, homepage probes
|
||
green. Documented the verified rootless identity model — container root
|
||
(PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0
|
||
(`/proc/<pid>/uid_map` = `0 1000 1`, `1 100000 65536`), so data roots are
|
||
`1000:1000` and **`userns_mode` must never be added to compose services**
|
||
(podman-compose 1.5.0 runs services in pods; `--userns` + `--pod` is
|
||
rejected). FrankBook-Pro differences noted: no gluetun egress gate on the
|
||
Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user
|
||
section and the rest in admin.
|
||
## 2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)
|
||
- Scope: every page under `/serverAppConfigs-docs/` re-verified against live
|
||
hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits
|
||
only — no pages created, renamed, or deleted. Per the deploy planner, a
|
||
docs-only change is a deploy no-op.
|
||
- Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
|
||
+ gluetun run on **FrankBook-Pro** (units installed to
|
||
`/home/core/.config/containers/systemd/` via the frankbook-pro deploy
|
||
handler; `PUID=0/PGID=0`, data roots `/Volumes/COMPLETE/<svc>-data:/config`;
|
||
qB pinned `4.6.7` sharing gluetun's netns with 8080 + 6881 published on the
|
||
gluetun unit; readarr pinned `amd64-1.0.980`; no vhosts — direct IP:port,
|
||
exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080
|
||
guarded), Calibre-Web (`0.6.27-ls397` + universal-calibre mod; no standalone
|
||
calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
|
||
- Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 →
|
||
internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana,
|
||
prometheus, loki, seerr) route post-auth to container-name upstreams;
|
||
exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP,
|
||
ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole
|
||
basicauth edge → :8083).
|
||
- `Gitea.md`: data roots + secrets corrected to the live unit paths
|
||
(`/home/dank/Containers/gitea/{data,postgres,env}`); UID gotcha corrected —
|
||
the units carry no `User=`/UserNS override, so container uid 1000 is literally
|
||
host uid 1000 (no subuid offset for these containers).
|
||
- Dedupe: `Prowlarr.md`/`prowlarr.md` collapsed into one canonical lowercase
|
||
page via rename (git history retains both versions; index link unchanged).
|
||
- LLM docs: archive pages (`DFlash DSpark Speculative Decoding`,
|
||
`ling-3.0-flash-trial`) reworded from current-tense to dated historical
|
||
framing with pointers to the active runbook [[kingjones-strix-rocmfp4]];
|
||
LLM `index.md` + `deployment.md` fleet tables and health-gate text updated to
|
||
the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui
|
||
abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771,
|
||
health `/v1/version` — FLM has no `/health` route).
|
||
- Suite green after the sweep: `./tests/run.sh` → 233 pass / 0 fail, including
|
||
the docs-completeness seam that requires every page to stay index-linked.
|
||
## 2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs
|
||
- WireGuard credentials provisioned (second Proton config, server `PA-12`,
|
||
endpoint `51.179.177.2:51820`) → `/home/core/Containers/gluetun/env`
|
||
(0600, VM-local); `deploy.sh frankbook-pro --service gluetun --service
|
||
qbittorrent` brought both up; egress verified (Proton PA exit, ~19 MB/s
|
||
on a live torrent). The `.conf` stays in `~/Downloads`; the private key
|
||
never enters the repo.
|
||
- gvproxy binds wildcard `*:6881/udp` (Go sets `SO_REUSEADDR` on TCP but not
|
||
UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and
|
||
were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path.
|
||
macOS application firewall still filters gvproxy inbound from non-local
|
||
sources (stealth mode) — cross-host 8080 needs `socketfilterfw` allow.
|
||
- qB wiring: first-seed bug (hash under legacy `WebUI\Password` key, ignored
|
||
by qB ≥ 4.2) fixed live (`WebUI\Password_PBKDF2`) and in
|
||
`scripts/deploy-remote.sh`; `DefaultSavePath` repointed to
|
||
`/Volumes/COMPLETE/Finished Torrents` — qB 5.2 API adds with a category
|
||
do not inherit the category save path (no auto-TMM); download clients
|
||
created in all four arrs (sonarr/radarr/lidarr via API, tests pass;
|
||
readarr by direct `readarr.db` insert — its EOL validator rejects qB 5's
|
||
`204` login as auth failure); Prowlarr Applications sync sonarr/radarr/
|
||
lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema).
|
||
Legacy `Prowlarr-*` indexer rows (hostname `prowlarr:9696`, unresolvable
|
||
on the Mac default network) deleted. E2E: Sintel fetched + downloaded
|
||
through the tunnel at ~19 MB/s, completed into `Finished Torrents`.
|
||
- Docs: [[FrankBook-Pro]] rewritten to the active-tunnel reality.
|
||
|
||
|
||
## 2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config
|
||
- RDAP: `149.102.254.77` = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle
|
||
hosting partner); Dankclaw's tunnel v6 `2a07:b944::2:2` sits in
|
||
CH-PROTONVPN-20160719; observed exit IP `149.22.88.19` = CDNEXT-SEA-CG.
|
||
The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is
|
||
no peer-registration step on any box of ours (full 65535-port scan: no
|
||
SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770,
|
||
8443 - Proton's protocol-stealth port map, no admin surface).
|
||
- Mac gate corrected: provision a **second** Proton WG config (dashboard,
|
||
named e.g. `frankbook`) - one config = one keypair; two hosts on
|
||
Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values
|
||
paste straight into the gluetun `custom` env. Earlier prepared keypair on
|
||
dankclaw (`~/frankbook-wg/`) retired to `~/.AGENTTRASH/` (Proton generates
|
||
the keypair per config; no BYO pubkey).
|
||
- [[FrankBook-Pro]] provisioning section rewritten.
|
||
|
||
## 2026-09-01 — Jellyfin authentication moved to Authentik LDAP
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` (`4ad85fa`,
|
||
digest-pinned in `f4ef2b8`) — new `authentik-outpost-ldap.service` on
|
||
Dankclaw serving LDAP :389/:636 for Jellyfin. Image
|
||
`ghcr.io/goauthentik/ldap` pinned by **amd64 single-platform digest**:
|
||
podman 5.7.0 refuses the tag because it resolves to a nested manifest
|
||
list ("unexpectedly received a manifest list instead of a manifest for a
|
||
single image") — resolve via `podman manifest inspect | jq` and pin.
|
||
- Authentik objects provisioned via `ak shell` ORM (live, idempotent
|
||
get_or_create): LDAP provider `Jellyfin LDAP`, service account
|
||
`svc-jellyfin-ldap` + bind token, groups `jellyfin_users` /
|
||
`jellyfin_admins`, application `jellyfin`.
|
||
- `/scripts/deploy-remote.sh` (`691da47`) — `authentik-outpost-ldap` added
|
||
to `service_is_allowed` (the deploy runner has its own allowlist beyond
|
||
deploy-plan.py's).
|
||
- Two authentik-side rules proven by live bind tests (`ldapsearch` through
|
||
the published port):
|
||
1. Bind token **intent must be `verification`** — `app_password` tokens
|
||
are rejected by the authorization flow's password stage with
|
||
`Invalid password`.
|
||
2. Search requires the **`search_full_directory`** object permission on
|
||
the provider (granted via RBAC role `svc-jellyfin-ldap-search`);
|
||
without it the outpost returns only the bind user, so `memberOf`
|
||
filters match zero entries.
|
||
- Jellyfin side (host-only, no repo change): LDAP-Auth **23.0.0.0**
|
||
installed from the GitHub releases zip (the `repo.jellyfin.org` stable
|
||
manifest path 404s), config XML at
|
||
`config/plugins/configurations/LDAP-Auth.xml`, username attribute `cn`
|
||
(authentik 2026.x maps `uid` to an opaque hash), and — the big one —
|
||
**`LdapSearchAttributes` is comma-separated** (`Split(',')` in the plugin
|
||
source; semicolons silently produce "Found no users matching X").
|
||
End-to-end verified with a throwaway `jellyfin_users` member
|
||
(`AuthenticateByName` 200, non-admin policy), then removed from both
|
||
authentik and the Jellyfin DB directly (its admin-token cleanup 401'd —
|
||
the admin password in the session archive was OCR-corrupted; verified
|
||
offline against the stored PBKDF2-SHA512 hash that the real bootstrap
|
||
value still authenticates, unchanged).
|
||
- Docs: [[Authentik]] gained the LDAP-outpost section (objects table +
|
||
rules), [[Jellyfin]] gained the LDAP-Auth plugin section. Docs-only
|
||
commit → deploy no-op.
|
||
|
||
## 2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh`,
|
||
`/scripts/frank-small.service` (`af6cd72` → `34ab6b4` → reverted
|
||
in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both
|
||
live-verified as loader failures and reverted the same day to the Huihui
|
||
FP8 checkpoint:
|
||
- `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel):
|
||
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the
|
||
checkpoint's expert tensors do not follow the
|
||
`mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider
|
||
scans for.
|
||
- `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8`
|
||
(compressed-tensors block-fp8 128×128, standard naming): experts
|
||
resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale`
|
||
directly — block scales do not broadcast →
|
||
`RuntimeError: The size of tensor a (2048) must match ...`.
|
||
- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no
|
||
outage); service was back on Huihui within the hour and re-verified with
|
||
a real completion (reasoning parses; `17*23 → 391`, natural stop).
|
||
- `/serverAppConfigs-docs/LLM/freetoken-frank-small.md` — quant
|
||
matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both
|
||
compressed-tensors shapes documented with their exact errors.
|
||
- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported
|
||
"healthy after approximately 6s" for BOTH failed loads — the health gate
|
||
passes while the doomed worker is still starting. A model swap is only
|
||
proven by the journal (Uvicorn startup) + a real completion.
|
||
- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8`
|
||
(37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8`
|
||
(35 GB).
|
||
|
||
## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated
|
||
- **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every
|
||
bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,
|
||
Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule
|
||
exists for. Recovery doubled as the Phase 1 cutover.
|
||
- **Phase 1 (dankserver)**: six quadlet units shipped
|
||
(`dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/`) + first-cutover
|
||
deploy handlers (retire bare containers → rescue RomM's anonymous `/romm`
|
||
volume → merge-move `/opt/<svc>` into `/home/dank/Containers/<svc>` →
|
||
ordered start with health gates). All six verified: units `active`,
|
||
`is-enabled=generated` (wants in the generator dir), ports 8085→8080,
|
||
8096, 8083, 3923 answering, jellyfin holds `/dev/dri` render nodes,
|
||
entropy binds provably ro except copyparty's `/w`. MariaDB continuity
|
||
proven via the real `dank` admin row (created 2026-08-31), not a fresh
|
||
bootstrap. Stale compose-era `container-*.service` generate-systemd units
|
||
retired to `~/.AGENTTRASH/20260901-stale-generate-systemd`.
|
||
- **Deployer findings fixed** (each caught live, each with a new seam):
|
||
v1 generator rejects `Command=` (→ `Exec=`), `Device=` (→ `AddDevice=`,
|
||
whole-unit rejection is silent), and splits spaces in `HealthCmd=`
|
||
(→ omitted for copyparty); `migrate_data_root` must merge into pre-seeded
|
||
destinations and rename mapped-subuid roots (100998) with `sudo mv`
|
||
(numeric-preserving, `keep` mode skips the chown — chowning mapped data
|
||
breaks the in-container users); quadlet unit dir must be created on a host
|
||
that never had quadlet; missing `PublishPort` = app answers inside the
|
||
netns only (copyparty 3923); seeded env files must carry **compose-provided
|
||
vars only** — the image-baked `HOME=/root` in a `[Service]
|
||
EnvironmentFile=` kills the podman client itself.
|
||
- **Phase 2 (dankclaw)**: data roots of gitea, authentik, hermes trios moved
|
||
to `/home/dank/Containers/<svc>/` while running (rename is inode-bound —
|
||
zero downtime), units repointed, redeployed through `deploy.sh --service`;
|
||
auth chain verified end-to-end (outpost ping → 302 to flow).
|
||
`deploy_hermes` ordering bug fixed: generated units must be reloaded
|
||
**after** unit files are copied, not before (stale ExecStart crash-looped
|
||
the trio on the old path).
|
||
- **Docs**: [[RomM]], [[Jellyfin]], [[Calibre-Web]], [[Copyparty]] updated to
|
||
the quadlet reality; [[Storage Policy]] now documents the Containers layout
|
||
+ the two ownership regimes; [[Deployment Flow]] service-manager table
|
||
corrected (dankserver media is no longer "bare containers, inventory only").
|
||
|
||
## 2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause
|
||
- **Deployed** `frank-npu` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`,
|
||
dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626;
|
||
primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy
|
||
crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`.
|
||
- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19
|
||
whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a
|
||
perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU
|
||
groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The
|
||
same root cause explains the `amdgpu ... no vmid pasid mapping supported`
|
||
kernel line — a consequence, not a GPU fault.
|
||
- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps
|
||
the iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||
frank-large — while enabling IOMMU hardware + PASID, which SVA needs;
|
||
PASID domains are paged regardless of the default-domain type). Post-boot:
|
||
30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`,
|
||
`NRestarts=0` since boot.
|
||
- **Live-verified behavior**: whole-registry serving with per-request routing by
|
||
exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved
|
||
end-to-end (generated red PNG as `image_url` data-URI → correct color answer,
|
||
279 prompt tokens consumed); audio tower proven with a control pair (440 Hz
|
||
tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only
|
||
control). Health probe is `/v1/version` — FLM has no `/health` route.
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`,
|
||
`/scripts/frank-npu.service` + deployer allowlist entries
|
||
(`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent
|
||
(`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`,
|
||
`LimitMEMLOCK=infinity` — FLM requires unlimited memlock).
|
||
- **Docs**: new runbook [[frank-npu]] (linked from both indexes),
|
||
AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard
|
||
requirement. Client wiring: pi `models.json` registers the endpoint under the
|
||
Dankclaw provider with the exact FLM registry tag as model id.
|
||
|
||
## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
|
||
- **Dankclaw / frank-large**: swapped to
|
||
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's
|
||
pinned `kingjones30/ROCmFPX` runtime (built at
|
||
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF`
|
||
— npm/HF-bucket asset provisioning aborts the link at 98% otherwise).
|
||
On-host 113.47 GiB file verified byte-identical to the HF LFS oid
|
||
(no re-download). Kept our args (401408×2, ub 512, fa on, fit off);
|
||
fa-on works UNPATCHED on this model — the QSA F32 assert was
|
||
agentionai-specific. No spec decode (card ships none; MTP heads
|
||
unloadable). Validated live: ~225 s load, health 200, reasoning
|
||
completions, real 111k-token prefill at 213–232 t/s. Deployed via
|
||
git flow; manual-process race during validation caught and fixed
|
||
(unit was crash-looping on a bound port while its health check
|
||
passed — check MainPID, not just /health).
|
||
- **Dankserver / frank-small**: swapped the model within
|
||
FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
|
||
(native block-fp8, same loader path and pool plan; 35 GB downloaded
|
||
via `hf`, official checkpoint retained for rollback). Learned:
|
||
FreeToken's chat API 422s without an explicit `model` field. ~40
|
||
tok/s verified post-deploy. Both `.service` Descriptions refreshed.
|
||
- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook
|
||
(runtime build + swap procedure + validation), [[agentionai-fast-imatrix]]
|
||
marked RETIRED (rollback record), [[freetoken-frank-small]]
|
||
model sections updated, LLM index + AGENTS fleet tables refreshed.
|
||
|
||
## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted
|
||
- **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api`
|
||
(Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),
|
||
auth = the Prowlarr **global** key. The previously hand-rolled
|
||
`/torznab/{id}/feed` paths were Forms-auth catch-alls returning
|
||
`302 → /login` — which the arrs XML-parsed as "unexpected token '/'".
|
||
A source-verified librarian answer settled it in one shot.
|
||
- **Re-pointed all four arr indexer rows** (Radarr→YTS id7,
|
||
Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API;
|
||
every PUT passed the app's live connection validator (202); RSS enabled.
|
||
- **TV + music chains proven** by live search: Sonarr returned 76 current
|
||
releases via Prowlarr-Nyaa; Lidarr initially logged `0 active indexers`
|
||
(new rows default all-off) — enabling the search flags yielded 18
|
||
releases for YOASOBI – THE BOOK.
|
||
- **First real grab**: `MoviesSearch` → 3 YTS releases → *The Dark Knight
|
||
(2008) 2160p* (7.5 GiB) → qBittorrent 100 % → **imported**
|
||
(`hasFile: true`, 8,078,103,833 B at `/movies/The Dark Knight (2008)`,
|
||
hardlinked from the shared `/mnt/entropy` filesystem), queue empty.
|
||
- **Prowlarr Application auto-sync found broken by version skew**
|
||
(`Unable to cast object of type 'NzbDrone.Core.Thing…'` against Radarr
|
||
6.3) — the manual rows are the integration; documented.
|
||
- **qBittorrent permanent credentials installed**: the WebUI temp password
|
||
rotated on every boot (it had silently broken the arr clients once this
|
||
session). qb5 removed the API password endpoint and the lscr image has
|
||
no env support — wrote `WebUI\Password_PBKDF2` (source-verified
|
||
`base64(salt):base64(PBKDF2-HMAC-SHA512/100k)` form) into the conf with
|
||
the container stopped; login survives `podman restart` (204). All arr
|
||
download clients updated to it; value kept host-local in
|
||
`/home/dank/.qbpass` (0600), never in git.
|
||
- **Readarr parked with reason**: abandoned upstream; its qB client cannot
|
||
authenticate against qB 5 while three sibling apps pass with identical
|
||
credentials. Indexer row healthy; books pipeline awaits a successor.
|
||
- **Docs**: rewrote `Dankserver/prowlarr.md` (live wiring + six verified
|
||
gotchas incl. the joiner bug and the 302-masquerading-as-auth failure)
|
||
and `Dankserver/qbittorrent.md` (persistent-credential recipe, Referer
|
||
CSRF note).
|
||
|
||
## 2026-08-29 (late) — guarded-host sign-in fixed end to end
|
||
- **Diagnosed** both user symptoms: `Invalid next URL` = the flow being entered with an absolute cross-host `next=` (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had **no `auth` A record** (added via Porkbun API, `auth A 71.227.227.172 TTL 300`).
|
||
- **Disproved meta-header routing** live (full meta-header set still 404s; matching `Host` alone answers). The 2026.8 embedded outpost routes solely on the original `Host` vs each proxy provider's `external_host`.
|
||
- **Provisioned** proxy providers + applications for `search`, `hermes`, `prometheus`, `romm`, `files` (pks 2–6) via ORM, attached to the embedded outpost; `internal_host = http://caddy:8017` everywhere.
|
||
- **Wedge found + fixed live:** providers created with null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch (`Failed to fetch outpost from API, retrying in 3 seconds` forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
|
||
- `/dankclaw/Caddyfile` (`29f9a86`) — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; deleted the per-vhost redirect/flow design and stale comments.
|
||
- `/dankclaw/anubis/policy.json` (`2a72efa`) — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance; app paths remain challenged.
|
||
- **Gotcha:** the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile). Suite green (`0` not-ok).
|
||
- `/docs/Infrastructure/Authentik.md`, `/docs/Infrastructure/Anubis.md`, `/docs/Architecture/Network Architecture.md` (`ee1e038`) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
|
||
- **Verified** by headless JS client: `https://search.franksdank.cloud/` → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 `state` JWT. Credential submission is the user's step.
|
||
|
||
## 2026-09-14
|
||
- **Creation**: [[Hermes-WebUI]] — new `hermes-webui` compose service on Dankclaw (the Hermex server): runs `hermes-webui`'s `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data` with the gateway (identical rootless Podman uid mapping), publishes `127.0.0.1:8787` only, password-gated via `HERMES_WEBUI_PASSWORD` in the host `.env`.
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — service definition and password placeholder; container-internal bind is `0.0.0.0` (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
|
||
- `/tests/hermes_webui.bats` — new seam: foreground entrypoint, loopback publish + auth, shared volume, `/health` healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
|
||
- Remote access pattern documented: iPhone Termius local forward `127.0.0.1:8787 -> 127.0.0.1:8787` over the existing router port forward to Dankclaw `:22`; Hermex points at `http://localhost:8787`. No Tailscale, no Cloudflare.
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts now proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
|
||
- `/docs/Infrastructure/Caddy.md` — Documented the Plex TLS-upstream requirement and the `/web/` vs `/` distinction.
|
||
- `/tests/caddyfile.bats` — Added a test pinning the TLS upstream transport on both Plex vhosts.
|
||
- `/docs/Dankclaw/Gitea.md` — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
|
||
- `/docs/Dankclaw/Dankclaw Stack.md` — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
|
||
- `/docs/Architecture/Deployment Flow.md` — Replaced the split-proxy safety description with the canonical Caddy ownership model.
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
|
||
- `/scripts/deploy-plan.py` — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
|
||
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
|
||
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
|
||
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
|
||
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
|
||
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
|
||
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start — `is-enabled` reports `generated` for all healthy quadlets, and an explicit `systemctl enable` fails on generated units (learned live during the deploy). `deploy_pihole` enforces the sysctl prerequisites.
|
||
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
|
||
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.
|
||
|
||
## 2026-09-01
|
||
- dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to `/home/dank/Containers/<svc>`, `/opt/{romm,jellyfin,calibre-web,copyparty}` retired, all four endpoints answering, legacy `container-*` units gone. Dankserver deploy-state for the six services records deployed revisions through `1a257c0`.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 `@ByteArray` WebUI password (plaintext is ignored by qB ≥ 4.2), sets `HostHeaderValidation=false` + `ServerDomains=*` for the socat forwarder path, and pre-creates the four categories with `save_path=/Volumes/COMPLETE/Finished Torrents`.
|
||
- `frankbook-pro` — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
|
||
- jellyfin GPU posture on dankserver: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` in the unit (parity with pre-migration reality — no `/dev/nvidia*` in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` (new), `/docs/index.md` — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider `seerr-proxy` (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
|
||
- `/scripts/deploy-remote.sh` — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
|
||
- `/tests/seerr_quadlet.bats` — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
|
||
- `/serverAppConfigs-docs/Dankclaw/Seerr.md`, `/serverAppConfigs-docs/index.md` — service doc + index entry.
|
||
- Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
|
||
- Authentik outpost claim cache fixed (root cause of the persistent `/admin` 403 despite homepage-admins membership): the outpost serves per-browser-session claims from `authentik_providers_proxy_proxysession` until row expiry — akadmin's live row predated the group add and carried `["authentik Admins"]` only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in `/serverAppConfigs-docs/Infrastructure/Authentik.md` (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
|
||
- WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway `10.88.0.1:8080`; Prowlarr's client created via its `/downloadclient/schema` template (hand-minimal body 400s on create-time test).
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — pinned `lscr.io/linuxserver/qbittorrent:4.6.7`: `latest` had resolved to qB 5.x, whose `204` empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy `200` + `"Ok."`. Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login `invalid credentials`; rewriting the same line fresh (same password, 100k iterations) fixed it.
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into `Finished Torrents`, deleted with data).
|
||
- `/scripts/deploy.sh` + `/scripts/deploy-plan.py` + `/tests/dankserver_quadlet.bats` — committed earlier today (`b206f47`): dankserver quadlet standardization units/handlers/routing.
|
||
|
||
## 2026-09-04 — Sonarr moved to the admin side; internal-only by construction
|
||
- Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an
|
||
**admin-side** service and must be locally reachable only. The homepage
|
||
registry flipped its card from the user category to **admin** and repointed
|
||
the link at `http://192.168.0.112:8989` (LAN-direct). Users keep requesting
|
||
TV through Seerr, which stays on the user side behind the guard chain.
|
||
- Exposure closure: the unguarded public vhost `sonarr.franksdank.cloud`
|
||
(direct `reverse_proxy 192.168.0.112:8989`, no auth) and its post-auth
|
||
`@sonarr` handler were retired from the Caddyfile; the Pi-hole addn-hosts A
|
||
record for the name was dropped with it. No public Porkbun record ever
|
||
existed (NXDOMAIN at the authoritative zone), so after this change there is
|
||
no WAN path to Sonarr at all — manual DNS could previously reach the vhost,
|
||
which served the zero-auth UI to anyone who resolved the name manually at
|
||
the public edge; the vhost (added in `e0bade6`, 2026-09-04) is retired.
|