126 lines
27 KiB
Markdown
126 lines
27 KiB
Markdown
# Bundle Change Log
|
||
|
||
## 2026-09-25
|
||
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
|
||
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
|
||
|
||
## 2026-09-22
|
||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-small-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
|
||
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
|
||
|
||
## 2026-09-21
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — frank-medium cut back to the MXFP4 checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`); `KV_DTYPE` became a quadlet tunable, set `auto` (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
|
||
- `/dankserver/calibre-server/quadlet/calibre-server.container` — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, `--auth-mode=basic` pinned, internal port 8090.
|
||
- `/serverAppConfigs-docs/Dankserver/Calibre.md` — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
|
||
- `/scripts/serve-frank-large.sh`, `/scripts/frank-large.env`, `/scripts/frank-large.service`, `/scripts/download-frank-large-unsloth.sh` — frank-large (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
|
||
- `/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` — New runbook; `frank-large-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
|
||
- `/shared/prometheus/prometheus.yml` — `node-dankclaw` target fixed to `host.containers.internal:9100` (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); `Infrastructure/Prometheus.md` targets table + `tests/observability_quadlet.bats` contract updated.
|
||
|
||
## 2026-09-20
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
|
||
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
|
||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
|
||
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
|
||
|
||
## 2026-09-19
|
||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
|
||
- `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook.
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
|
||
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
|
||
|
||
## 2026-09-18
|
||
- `/scripts/frank-medium.env` — `CACHE_RAM` 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; `tests/frank_medium_unit.bats` seam added; `--ctx-checkpoints 0` confirmed a deliberate deviation (build default is 32) (1acca43).
|
||
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
|
||
|
||
## 2026-09-17
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||
- `/scripts/frank-small.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||
- `/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` — New ACTIVE runbook; `frank-small-qwen38-gsq-rco.md` retired as rollback target.
|
||
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
|
||
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
|
||
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
|
||
|
||
## 2026-09-16
|
||
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
|
||
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||
- `/scripts/frank-small.env`, `/scripts/frank-small.service`, `/scripts/serve-frank-small.sh` — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
|
||
- `/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
|
||
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
|
||
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
|
||
|
||
## 2026-09-15
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh`, `/scripts/frank-medium.service` — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (`retire_root_llm_small()` archived the CUDA-era unit).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` — New runbook; `freetoken-frank-small.md` and `kingjones-q8agent-rocmfpx.md` marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
|
||
- Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the `ld.so.conf` hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`).
|
||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
|
||
- `/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` — New runbook with the full evidence chain.
|
||
|
||
## 2026-09-14
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — New `hermes-webui` service (Hermex): `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data`, loopback-only publish 127.0.0.1:8787, password-gated via `HERMES_WEBUI_PASSWORD`.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories.
|
||
- `/tests/hermes_webui.bats` — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
|
||
- `/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md` — New page; remote access = Termius local forward over the existing router :22 port forward.
|
||
|
||
## 2026-09-12
|
||
- `/dankclaw/marreta/quadlet/` — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; `NGINX_HTTP_PORT=8085` (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
|
||
- `/tests/marreta_quadlet.bats` — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
|
||
- `/serverAppConfigs-docs/Dankclaw/Marreta.md` — New page (incl. the `base64:`-prefixed APP_KEY gotcha).
|
||
- Live — authentik ProxyProvider `marreta` (pk 13) + Application link: the outpost's config API filters `application__isnull=False`, so a provider with no linked Application is never loaded (the login-wall root cause); a null `invalidation_flow` crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec `16-marreta.spec.ts`.
|
||
|
||
## 2026-09-04
|
||
- `/frankbook-pro/` — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, `/opt` config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed `10.88.0.1:8080` → `127.0.0.1:8080`.
|
||
- `/tests/frankbook_quadlet.bats` — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
|
||
- `/dankserver/` — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned `/opt/<svc>/config` dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and `userns_mode` must NEVER be added (podman-compose runs services in pods; `--userns` + `--pod` is rejected).
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
|
||
- `/tests/e2e/` — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).
|
||
|
||
## 2026-09-03
|
||
- `/serverAppConfigs-docs/` — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, `Prowlarr.md`/`prowlarr.md` deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.
|
||
|
||
## 2026-09-01
|
||
- `/frankbook-pro/gluetun/`, `/frankbook-pro/qbittorrent/` — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (`WebUI\Password_PBKDF2`, not the legacy `WebUI\Password` key qB ≥ 4.2 ignores); `DefaultSavePath` → `/Volumes/COMPLETE/Finished Torrents`; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
|
||
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
|
||
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + `HostHeaderValidation=false` + the four categories; full 7-service deploy run.
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — qB pinned `4.6.7` (5.x's `204` login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
|
||
- `/serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md`, `/serverAppConfigs-docs/Architecture/Storage Policy.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.
|
||
|
||
## 2026-08-30
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
|
||
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
|
||
- `/scripts/frank-small.env` — frank-small (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
|
||
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
|
||
|
||
## 2026-08-29
|
||
- `/dankclaw/Caddyfile` — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
|
||
- `/dankclaw/anubis/policy.json` — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile).
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Infrastructure/Anubis.md`, `/serverAppConfigs-docs/Architecture/Network Architecture.md` — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original `Host` vs each provider's `external_host` (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch; the public zone gained the `auth` A record via the Porkbun API (ee1e038).
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
|
||
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
|
||
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|