docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
@@ -12,24 +12,31 @@ timestamp: 2026-10-05

 Two ready-to-load text-to-image workflows for the Qwen-Image 2.1 payload
 already on Dankclaw, both proven end-to-end on 2026-10-05 by POSTing to the
-live API on port 8188 and verifying the rendered PNG. The repo copies are the
+live API on port 8188 and verifying the rendered PNG (the renders are still on
+disk: `qwen21-art_00001_.png` 22:41 and `qwen21-text_00001_.png` 22:44,
+`ls -l /home/dank/comfyui-data/output/` 2026-10-18). The repo copies are the
 source of truth; the host copies under `/home/dank/comfyui-data/workflows/`
-are for convenience.
+are for convenience and are byte-identical today (`md5sum` matches both files,
+2026-10-18).

 ## What is on disk

+Sizes are GiB, measured `ls -l` on the host 2026-10-18 (all files dated
+2026-09-21).
+
 | Path (under `/home/dank/comfyui-data/models/`) | Size | Role |
 |---|---|---|
-| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
-| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 14 GB | Full-precision fallback |
-| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB | GGUF alternative (needs ComfyUI-GGUF custom node) |
-| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
-| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.8 GB | Mid quant |
-| `text_encoders/qwen3vl_8b_bf16.safetensors` | 17 GB | Full precision |
-| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB | 2.1 VAE (used) |
-| `vae/qwen_image_vae.safetensors` | 243 MB | Smaller VAE |
+| `diffusion_models/qwen_image_2.1_int8_convrot.safetensors` | 6.8 GB (7,256,783,064 B) | **The one the workflows use** — int8-convrot quant, ComfyUI-core native |
+| `diffusion_models/qwen_image_2.1_bf16.safetensors` | 13.3 GB (14,230,280,616 B) | Full-precision fallback |
+| `diffusion_models/qwen-image-2.1-Q8_0.gguf` (+ `-arch`) | 7.1 GB each (7,591,551,648 / 7,591,551,680 B) | GGUF alternative (needs ComfyUI-GGUF custom node, present in `custom_nodes/`) |
+| `text_encoders/qwen3vl_8b_w4a8.safetensors` | 5.9 GB (6,312,105,364 B) | **The one the workflows use** — 4-bit-weights/8-bit-act Qwen3-VL 8B encoder |
+| `text_encoders/qwen3vl_8b_int8_convrot.safetensors` | 8.7 GB (9,350,798,360 B) | Mid quant |
+| `text_encoders/qwen3vl_8b_bf16.safetensors` | 16.3 GB (17,534,334,616 B) | Full precision |
+| `vae/qwen_image_2.1_vae_bf16.safetensors` | 645 MB (675,509,688 B) | 2.1 VAE (used) |
+| `vae/qwen_image_vae.safetensors` | 243 MB (253,806,246 B) | Smaller VAE |

-No LoRAs are on disk yet — see [Recommended additions](#recommended-additions).
+No LoRAs are on disk yet, and `models/lora/` / `models/controlnet/` do not
+exist — see [Recommended additions](#recommended-additions).

 ## The workflows

@@ -45,7 +52,13 @@ EmptySD3LatentImage ─────────┴─ KSampler ─ VAEDecode ─
 ```

 **Common sampler settings:** `cfg 1.0` (Qwen-Image 2.1 expects CFG-free
-sampling), `euler` + `simple`, seed fixed in the file.
+sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed
+fixed in the file (`1930` in the art workflow, `2026` in the text one) —
+read from `dankclaw/comfyui/workflows/*.json` node `9`.
+
+Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`,
+`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`,
+`SaveImage` (ids 3–11).

 | Workflow | File | Shape | Steps | Made for |
 |---|---|---|---|---|
@@ -68,8 +81,9 @@ prompt is written for it:

 Verified render: `SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS` + `DANKCLAW
 DANKSERVER EST 2026` came back crisp and correctly spelled at 832×1216
-(`qwen21-text_00001_.png`, 953 KB). One caveat from the same render: a
-shorter secondary line is where misspellings creep in — proofread the first
+(`qwen21-text_00001_.png`, 953,254 B — still on disk, `ls -l
+/home/dank/comfyui-data/output/` 2026-10-18). One caveat from the same render:
+a shorter secondary line is where misspellings creep in — proofread the first
 render before shipping.

 ## Driving it from the API
@@ -89,20 +103,28 @@ Outputs land in `/home/dank/comfyui-data/output/`. The queue is FIFO — the
 two workflows above ran back-to-back unattended.

 **Note:** the container's UI-saved workflows live in the ephemeral
-`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — hence the
-host-side `workflows/` dir and the repo copy.
+`/opt/ComfyUI/user` layer and do NOT survive an image rebuild — the unit mounts
+only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
+(`podman inspect comfyui` 2026-10-18; `dankclaw/comfyui/quadlet/comfyui.container:48-52`)
+— hence the host-side `workflows/` dir and the repo copy.

 ## Recommended additions

-Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
-frank-zealot's). All repos verified 2026-10-05. None are on disk yet.
+Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
+reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
+2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
+all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`,
+`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`,
+`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`,
+`huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk
+yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18).

 | # | What | Repo | Grab | Size | Why |
 |---|---|---|---|---|---|
 | 1 | **Lightning 8-step LoRA** (T2I speed) | `lightx2v/Qwen-Image-Lightning` | `qwen_image_lightning_8steps_*.safetensors` (LoRA) | ~1–2 GB | 20 steps → 8 with a `LoraLoader` node; 2–3× faster renders on the iGPU, negligible quality loss |
 | 2 | **Qwen-Image-Edit-2511 GGUF** (image editing) | `unsloth/Qwen-Image-Edit-2511-GGUF` | Q4_K_M (~13.2 GB) or Q4_0 (~11.9 GB) | ~12–13 GB | Unlocks img2img "edit this photo" workflows — a capability class the fleet has zero of; Q4 keeps it iGPU-friendly |
-| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI |
-| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` CLIP type is already in this build's loader enum) |
+| 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) |
+| 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) |

 Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
 #3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
@@ -11,27 +11,31 @@ timestamp: 2026-09-16

 ComfyUI is a node-based AI image generation engine. Workflows are built from nodes that load models, sample latents, and post-process images; everything is scriptable through a web UI and an API.

-**Host:** Dankclaw — **Port:** `8188` published on the host; **no vhost and no
-guard chain** — ComfyUI is LAN-only by construction (the Caddyfile routes no
-name to it) — **Backend:** ROCm GPU (gfx1151) — **Docs:**
-https://github.com/Comfy-Org/ComfyUI
+**Host:** Dankclaw — **Port:** `0.0.0.0:8188` published, **no auth** (`ss -ltnp`
+2026-10-18) — **no vhost and no guard chain**: the Caddyfile routes no name to
+it (no `comfyui` or `8188` line anywhere in `dankclaw/Caddyfile`), so LAN access
+rides the Pi-hole A record `comfyui.franksdank.cloud`
+(`dankclaw/pihole/addn-hosts:37`) plus the raw port — the homepage card links
+`http://comfyui.franksdank.cloud:8188` and probes `127.0.0.1:8188`
+(`shared/homepage/services.json:519-521`) — **Backend:** ROCm GPU (gfx1151) —
+**Docs:** https://github.com/Comfy-Org/ComfyUI

 ## Facts

 | Item | Value |
 |------|-------|
-| Image | `comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui` |
-| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` |
-| Upstream latest | 0.37.0 (2026-09-21) — ships **native Qwen-Image 2.1** support (CORE-423, `comfy/ldm/qwen_image21/`). The image's custom GGUF patches may be unnecessary on the next rebuild; check before re-applying. |
-| Port | 8188 (host) → 8188 (container) |
-| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
-| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
-| Run | `./scripts/deploy.sh dankclaw --service comfyui` (Quadlet unit `comfyui.service`) |
-| Boot-persistent | Quadlet unit `dankclaw/comfyui/quadlet/comfyui.container` → `~/.config/containers/systemd/` |
+| Image | `localhost/comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui`, `Pull=never` (`dankclaw/comfyui/quadlet/comfyui.container:41-42`; `podman inspect comfyui` 2026-10-18) |
+| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` (`dankclaw/Containerfile.comfyui:9`) |
+| Upstream latest | 0.37.0 — the version the running container reports (`GET http://127.0.0.1:8188/system_stats`, 2026-10-18). The release date 2026-09-21 and its **native Qwen-Image 2.1** support (CORE-423, `comfy/ldm/qwen_image21/`) *(unverified 2026-10-18)* — the image's own startup patch still rewrites detection and VAE-decode code at every boot (`dankclaw/apply_qwen21_patch.py:4,23,37`, run by `dankclaw/comfyui-start.sh:3`), so check before re-applying on the next rebuild. |
+| Port | 8188 (host) → 8188 (container), published `0.0.0.0` (`dankclaw/comfyui/quadlet/comfyui.container:44`) |
+| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag — the unit runs `python main.py --listen 0.0.0.0 --port 8188` (`dankclaw/comfyui-start.sh:4`). `rocm-smi --showproductname` reports `GFX Version: gfx1151` (2026-10-18). |
+| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` (`dankclaw/comfyui/quadlet/comfyui.container:48-51`) |
+| Run | `./scripts/deploy.sh dankclaw --service comfyui` (Quadlet unit `comfyui.service`; handler `deploy_comfyui`, `scripts/deploy-remote.sh:649`) |
+| Boot-persistent | Quadlet unit `dankclaw/comfyui/quadlet/comfyui.container` → `~/.config/containers/systemd/`; live 2026-10-18 `systemctl --user is-enabled comfyui` → `generated`, `is-active` → `active` |

 ## Why the GPU works here

-Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S`, capability `(11, 5)`. See [[ComfyUI ROCm and Vulkan]] for the full verification.
+Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S Graphics` (`GET http://127.0.0.1:8188/system_stats` 2026-10-18: one device, `cuda:0 Radeon 8060S Graphics : native`, type `cuda`), capability `(11, 5)` (`dankclaw/Containerfile.comfyui:4-5`). See [[ComfyUI ROCm and Vulkan]] for the full verification.

 ## How the image is built

@@ -39,16 +43,22 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not

 ## GPU device access (rootless), and why the unit exists

-The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`) and uses
+The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`,
+`dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses
 `GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
 cannot map the group name into the user namespace. The companion udev rule
 `dankclaw/udev/99-comfyui-gpu-access.rules` keeps those nodes `MODE 0666`, which
 is what lets a **user-unit spawn** — carrying no supplementary groups — open
-them anyway. Verified inside the running container: `torch.cuda.is_available()`
-is `True` and the device is `Radeon 8060S Graphics`.
+them anyway. The rule is installed on the host
+(`/etc/udev/rules.d/99-comfyui-gpu-access.rules`, `ls` 2026-10-18) and the nodes
+are world-writable today: `crw-rw-rw- /dev/kfd`, `crw-rw-rw-
+/dev/dri/renderD128` (`ls -l` 2026-10-18). Verified through the running
+service: `/system_stats` reports the CUDA-type device, i.e.
+`torch.cuda.is_available()` is `True` and the device is `Radeon 8060S Graphics`.

-**Incident, 2026-09-16 — why this is a Quadlet now.** ComfyUI ran from
-`dankclaw/compose.yaml` through a hand-written `~/.config/systemd/user/
+**Incident, 2026-09-16 (retired era) — why this is a Quadlet now.** ComfyUI ran
+from `dankclaw/compose.yaml` (no longer in the repo) through a hand-written
+`~/.config/systemd/user/
 comfyui.service` that called `podman compose up -d`. Its container was created
 *before* the GPU declarations were added to the compose service, and
 `restart: unless-stopped` then kept restarting that stale container: no
@@ -58,7 +68,8 @@ fine; the container was old. A `--force-recreate` would have fixed it that day,
 but the class of bug needed the unit: one owner per container name, repo-tracked,
 and recreated from the source of truth on every deploy. The compose service is
 deleted; `deploy_comfyui` retires any compose-era unit into `~/.AGENTTRASH/`
-and removes a stale container before installing the unit.
+(`scripts/deploy-remote.sh:649,660`) and removes a stale container before
+installing the unit.

 ## Volumes

@@ -75,7 +86,10 @@ The ComfyUI code is baked into the image (not mounted). Rebuild the image to upd

 Custom nodes go in `/home/dank/comfyui-data/custom_nodes` (mounted at `/opt/ComfyUI/custom_nodes`). ComfyUI loads every subdirectory there on start. Because this path is a bind mount, it shadows anything baked into the image — so custom nodes are installed host-side, not in the Containerfile.

-**ComfyUI-Manager** is installed here. Install it with a one-time clone, then restart the container:
+**ComfyUI-Manager** is installed here, alongside **ComfyUI-GGUF** (`ls
+/home/dank/comfyui-data/custom_nodes` 2026-10-18 — the GGUF node's Python deps
+are baked into the image, `dankclaw/Containerfile.comfyui:29`). Install Manager
+with a one-time clone, then restart the container:

 ```bash
 git clone https://github.com/ltdrdata/ComfyUI-Manager \
@@ -83,19 +97,29 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager \
 systemctl --user restart comfyui
 ```

-ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` (returns a version like `V3.41`). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.
+ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` — it answers `V3.41` today (`curl http://127.0.0.1:8188/manager/version` 2026-10-18). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.

 ## Boot persistence

 The Quadlet unit carries `[Install] WantedBy=default.target` with
-`Restart=always`, so the generator wires it into `default.target` and it comes
-back on its own (user linger is on). Inspect it with
-`systemctl --user is-enabled comfyui.service` → `generated`; an explicit
-`systemctl --user enable` fails on generated units.
+`Restart=always` and `RestartSec=10`
+(`dankclaw/comfyui/quadlet/comfyui.container:66-70`), so the generator wires it
+into `default.target` and it comes back on its own (user linger is on:
+`loginctl show-user dank -p Linger` → `yes`). Inspect it with
+`systemctl --user is-enabled comfyui.service` → `generated`, `is-active` →
+`active` (both 2026-10-18); an explicit `systemctl --user enable` fails on
+generated units.

 ## Healthcheck

-The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device).
+The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/`
+2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats`
+reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The
+health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`,
+bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
+(`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs
+`http://127.0.0.1:8188/` inside the container, and the container reports
+`healthy` (`podman inspect comfyui` 2026-10-18).

 ## Gotchas

@@ -103,8 +127,8 @@ The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `
 - Do not add the `render` group by name. Rootless podman cannot map it; the udev rule plus `keep-groups` is the mechanism that works.
 - **A stale container outlives a compose edit.** If the GPU declarations change, recreate the container (`deploy.sh dankclaw --service comfyui` removes the old one) — `restart: unless-stopped` will happily keep an old config alive forever.
 - Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
- The base image is about 32 GB. A code update means a rebuild, because the code is baked in.
- The model collection starts empty. Add models to `/home/dank/comfyui-data/models` before generating.
+- The base image is about 32 GiB (`podman image inspect localhost/comfyui-rocm:gfx1151` → 33,765,227,471 B ≈ 31.4 GiB, 2026-10-18). A code update means a rebuild, because the code is baked in.
+- The empty-models note from the 2026-09-16 era is retired: `/home/dank/comfyui-data/models` now holds the Qwen-Image 2.1 payload — three `diffusion_models`, three `text_encoders`, two VAEs (~35 GB, `ls -l` 2026-10-18), listed in [[ComfyUI Qwen-Image Workflows]]. New model classes (LoRA, ControlNet) still need their directories created first: `models/lora` and `models/controlnet` do not exist yet.

 ## See Also

@@ -14,23 +14,24 @@ timestamp: 2026-08-26

 Gitea is a self-hosted Git service with issues, pull requests, and a web UI. On Dankclaw it runs as two Quadlet-managed containers in the rootless Podman session of user `dank`. The canonical Caddy on the same host reverse-proxies its web UI at `gitea.franksdank.cloud`.

-**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com
+**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`), git SSH on `2222` (`ss -ltnp` 2026-10-18) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com

 ## Containers

 | Container | Unit | Image | Purpose |
 |-----------|------|-------|---------|
-| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` | Database |
-| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` | Web UI, API, git (HTTP :3000, SSH :2222) |
+| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` (`dankclaw/gitea/quadlet/gitea-postgres.container:18`) | Database |
+| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` (`dankclaw/gitea/quadlet/gitea-server.container:22`) — answers `GET /api/v1/version` → `1.27.3` (2026-10-18) | Web UI, API, git (HTTP :3000, SSH :2222, `dankclaw/gitea/quadlet/gitea-server.container:32-33`) |

-The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name. The automatic `podman` default network has DNS disabled. Do not run the Gitea containers on it.
+The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name; `podman network ls` 2026-10-18 confirms `systemd-gitea dns=true` while the automatic `podman` default network is `dns=false`. Do not run the Gitea containers on it.

 ## Hosted repositories

 | Repository | Purpose |
 |------------|---------|
 | `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
-| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
+| `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) |
+| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* |

 The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)

@@ -57,9 +58,9 @@ graph LR

 ## Boot persistence

-User-level linger is enabled for `dank` (`loginctl enable-linger dank`). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target`. The podman generator applies it at boot; no symlink is needed.
+User-level linger is enabled for `dank` (`loginctl enable-linger dank`; `loginctl show-user dank -p Linger` → `yes`, 2026-10-18). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target` with `Restart=always` (`dankclaw/gitea/quadlet/gitea-server.container:41,43-44`; `gitea-postgres.container:28,30-31`); live 2026-10-18 both report `systemctl --user is-enabled` → `generated` and `is-active` → `active`. The podman generator applies the install at boot; no symlink is needed.

-The containers use the quadlet fresh-start model. A stop removes the container (`podman run --rm --replace`). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.
+The containers use the quadlet fresh-start model. A stop removes the container — the generated unit runs `podman run --name gitea-server --replace --rm …` with `ExecStopPost=-/usr/bin/podman rm -v -f -i gitea-server` (`systemctl --user cat gitea-server.service` 2026-10-18). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.

 ## Secrets

@@ -90,22 +91,26 @@ Per the live quadlet units (`Volume=` lines):

 | Container path | Host path | Contents |
 |----------------|-----------|----------|
-| `/data` | `/home/dank/Containers/gitea/data` | Git repositories, app.ini, attachments |
-| `/var/lib/postgresql/data` | `/home/dank/Containers/gitea/postgres` | PostgreSQL data |
+| `/data` | `/home/dank/Containers/gitea/data` | Git repositories, app.ini, attachments (`dankclaw/gitea/quadlet/gitea-server.container:34`) |
+| `/var/lib/postgresql/data` | `/home/dank/Containers/gitea/postgres` | PostgreSQL data (`dankclaw/gitea/quadlet/gitea-postgres.container:23`) |

 Back up `/home/dank/Containers/gitea/` (tar) or `pg_dump` the database. See the knowledge base page [[Gitea]].

 ## Gotchas

- **Data files are owned by `dank` (uid 1000), not root.** The quadlet units run
-  podman in the user manager with no `--user`/UserNS override, so container uid
-  1000 is literally dank's host uid — there is no subuid offset for these
-  containers. Edit files as the owning user with
-  `podman exec --user 1000 gitea-server ...`.
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
- **Port 3000, and one owner.** The live host port is 3000, published by the `gitea` Quadlet unit. The `dankclaw/compose.yaml` that mapped 3001 was **retired on 2026-09-16** (staged in `.AGENTTRASH/`) once an ownership audit showed every container on this host reports a `PODMAN_SYSTEMD_UNIT` label — a second composition path could only spawn a duplicate fighting a unit for a name. Do not reintroduce a compose service for Gitea or the proxy.
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
- **Canonical proxy.** `gitea.franksdank.cloud` is served by the canonical Caddy **unit** (`dankclaw/caddy/quadlet/caddy.container`, repo `dankclaw/Caddyfile`), which proxies to `192.168.0.143:3000`. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. If a legacy unit is ever installed again, the deployer stages it under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy.
+- **Gitea's uid 1000 is NOT dank's host uid.** The units run podman in the user
+  manager with no `UserNS=`/`--user` override (`podman inspect gitea-server
+  --format '{{json .HostConfig.IDMappings}}'` → `null`, i.e. the default rootless
+  map), so container uid 1000 (the `git` user) lands on host uid **100999**
+  (the first subuid plus 999): `ls -ln /home/dank/Containers/gitea/data` shows
+  `git/` and `gitea/` owned by `100999` while `ssh/`, created host-side, is
+  `1000` (`ls -ln` 2026-10-18). Edit app data through the container instead of
+  the host path: `podman exec --user 1000 gitea-server ...`.
+- **The host gets rebooted.** The 2026-08-16 incident below was an `unattended-upgrades` reboot; today `unattended-upgrades` is enabled but `Unattended-Upgrade::Automatic-Reboot` is not set anywhere in `/etc/apt/apt.conf.d/` (`apt-config dump` and `grep` 2026-10-18 — the key is present only commented-out at `50unattended-upgrades:94`). Do not rely on the trigger: the Quadlet units above are what make Gitea survive any reboot.
+- **Port 3000, and one owner.** The live host ports are 3000 (HTTP) and 2222 (git SSH), published by the `gitea-server` Quadlet unit (`dankclaw/gitea/quadlet/gitea-server.container:32-33`; `ss -ltnp` 2026-10-18). The `dankclaw/compose.yaml` that mapped 3001 was **retired on 2026-09-16** (staged in `.AGENTTRASH/`) once an ownership audit showed every container on this host reports a `PODMAN_SYSTEMD_UNIT` label — a second composition path could only spawn a duplicate fighting a unit for a name. Do not reintroduce a compose service for Gitea or the proxy.
+- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/` (`ls` 2026-10-18 shows the file there, 1,278 B). Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
+- **Canonical proxy, and it is exempt.** `gitea.franksdank.cloud` is served by the canonical Caddy **unit** (`dankclaw/caddy/quadlet/caddy.container`, repo `dankclaw/Caddyfile`), which proxies to `192.168.0.143:3000` and is on the **exempt** list — no Anubis challenge, no Authentik gate — because git clients cannot solve a JS proof; Gitea's own login is the only gate. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. If a legacy unit is ever installed again, the deployer stages it under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy (`scripts/deploy-remote.sh:347,365-370`).
+- **`/metrics` is off, so Prometheus cannot scrape Gitea.** `curl http://127.0.0.1:3000/metrics` → **404** and the container carries no metrics-related environment variable (`podman inspect gitea-server` filtered 2026-10-18; no `ENABLE_METRICS`/`PROMETHEUS` line in either quadlet). The `gitea` job (`shared/prometheus/prometheus.yml:46-49`, target `192.168.0.143:3000`, `metrics_path: /metrics`) is therefore DOWN with `server returned HTTP status 404 Not Found` (`/api/v1/targets` 2026-10-18). Gitea is covered by the homepage status probe, not by metrics.

 ## Incident: 2026-08-16 — down after unattended reboot

@@ -11,17 +11,17 @@ timestamp: 2026-08-28

 Gluetun is a lightweight VPN client container. It connects to a VPN provider (WireGuard or OpenVPN) and shares that tunnel with other containers that attach to its network namespace.

-**Host:** Dankclaw — **Docs:** https://github.com/qdm12/gluetun
+**Host:** Dankclaw — **Tunnel:** WireGuard (`VPN type: wireguard`, journal 2026-10-18) to ProtonVPN US-WA per `dankclaw/searxng/quadlet/gluetun.container:6` — **Exit:** `149.22.88.19`, United States, Washington, Seattle (`journalctl --user -u gluetun`, `[ip getter]`, 2026-10-18; the journal never names the provider, so the provider attribution is the repo's) — **Docs:** https://github.com/qdm12/gluetun

 ## Facts (from research, primary source)

 | Item | Value |
 |------|-------|
-| Image | `qmcgaw/gluetun:latest` |
-| Capabilities | `NET_ADMIN` required |
-| Device | `/dev/net/tun:/dev/net/tun` |
-| Ports published | searxng's 8081 → 8080 lives here |
-| Tunnel check | the `tun0` interface exists (the `:8000` server 404s on all paths) |
+| Image | `docker.io/qmcgaw/gluetun:latest` (`dankclaw/searxng/quadlet/gluetun.container:19`; live 2026-10-18). Tagged `:latest`, so a redeploy can move the version — pin it deliberately if that ever matters. |
+| Capabilities | `NET_ADMIN` required (`dankclaw/searxng/quadlet/gluetun.container:20`) |
+| Device | `/dev/net/tun:/dev/net/tun` (`dankclaw/searxng/quadlet/gluetun.container:21`) |
+| Ports published | Two, both on this container because it owns the netns: `8081 → 8080` (SearXNG) and `8082 → 8085` (Marreta) (`dankclaw/searxng/quadlet/gluetun.container:31-32`; `podman inspect gluetun --format '{{json .NetworkSettings.Ports}}'` and `ss -ltnp` 2026-10-18). Neither consumer container publishes anything. |
+| Tunnel check | the `tun0` interface exists — the boot config dump logs `Network interface: tun0` (`journalctl --user -u gluetun` 2026-10-18). The control server listens on `:8000` **inside the netns only** (`.NetworkSettings.Ports` shows `"8000/tcp": null`, 2026-10-18), so nothing on the host can reach it. |

 ## Required Environment

@@ -32,8 +32,8 @@ environment:
  WIREGUARD_PRIVATE_KEY: "${WIREGUARD_PRIVATE_KEY}" # from .env, never committed
 ```

- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it.
- Provider-specific variables come from the Gluetun wiki; set them in `.env`.
+- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it (https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/vpn.md). This deployment runs WireGuard (`WIREGUARD_*` variables, `dankclaw/searxng/quadlet/gluetun.container:24-30`).
+- Provider-specific variables come from the Gluetun wiki (https://github.com/qdm12/gluetun-wiki); set them in `.env`.

 ## Optional DNS (from research)

@@ -44,11 +44,13 @@ environment:
 | `DNS_CACHING` | `on` | cache queries |
 | `BLOCK_MALICIOUS` | `on` | block malicious hostnames |

+Names and defaults match the upstream wiki (https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/dns.md); live confirmation 2026-10-18: `[dns] ready and using DNS server with dot upstream resolvers` and `[dns] downloading hostnames and IP block lists`.
+
 Warning from the wiki: do NOT set `DNS_UPSTREAM_PLAIN_ADDRESSES` to a LAN/bridge address — DNS queries leak out of the tunnel.

 ## Attaching a Container

-The searxng unit reuses gluetun's network namespace:
+Three units reuse gluetun's network namespace — SearXNG and both Marreta containers:

 ```ini
 [Container]
@@ -56,11 +58,11 @@ ContainerName=searxng
 Network=gluetun.container
 ```

-`Network=gluetun.container` also adds a systemd dependency on gluetun, so searxng starts after the tunnel.
+`Network=gluetun.container` also adds a systemd dependency on gluetun, so the consumer starts after the tunnel (`dankclaw/searxng/quadlet/searxng.container:13-18`, `dankclaw/marreta/quadlet/marreta.container:29-34`, `dankclaw/marreta/quadlet/marreta-browser.container:20-25`). Live 2026-10-18 all three report `HostConfig.NetworkMode=container:03a75714…`, which is gluetun's own container id (`podman inspect gluetun --format '{{.Id}}'`) — one netns, four containers, ports published only by gluetun.

 ## Boot Persistence

-gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and user linger is on, so the tunnel starts at boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600). Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`.
+gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and `Restart=always` (`dankclaw/searxng/quadlet/gluetun.container:37,39-40`) and user linger is on, so the tunnel starts at boot — live 2026-10-18 `systemctl --user is-enabled gluetun` → `generated`, `is-active` → `active`, `NRestarts=0` since the 01:25 boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600 — `-rw------- dank:dank`, `ls -l` 2026-10-18). Deploy with `scripts/deploy.sh dankclaw --service gluetun` (`scripts/deploy-remote.sh:596-597`); `scripts/deploy-searxng-quadlet.sh dankclaw` is a compatibility wrapper that now forwards to `deploy.sh --service searxng`, so it does **not** deploy the tunnel.

 ## Verify the Tunnel

@@ -68,13 +70,14 @@ gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio).
 podman exec gluetun curl -s ifconfig.me
 ```

-The returned IP must differ from the host IP.
+The returned IP must differ from the host IP. A read-only alternative is the journal: gluetun logs its own egress at start, and on 2026-10-18 it logged `[ip getter] Public IP address is 149.22.88.19 (United States, Washington, Seattle …)` — not the host's address.

 ## Gotchas

- The container that holds the published port must be the one with the ports mapping (gluetun).
- `restart: always` on gluetun so the tunnel reconnects after VPN drops.
+- The container that holds the published port must be the one with the ports mapping (gluetun) — searxng, marreta and marreta_browser publish nothing of their own (`dankclaw/searxng/quadlet/searxng.container` and `dankclaw/marreta/quadlet/*.container` have no `PublishPort=`; `ss -ltnp` 2026-10-18 shows only gluetun's rootlessport on 8081/8082).
+- `Restart=always` on gluetun (`dankclaw/searxng/quadlet/gluetun.container:37`) so the tunnel reconnects after VPN drops.
 - When gluetun restarts, attached containers inherit the new network state; searxng may need a restart to re-resolve DNS.
+- The gluetun health-check probe cannot open ICMP sockets: it logs `listening for ICMP packets: not permitted: you can try adding NET_RAW capability to resolve this; permanently falling back to plain DNS` (journal 2026-10-18). The unit grants `NET_ADMIN` only; the fallback works, so this is expected noise, not a fault.

 ## See Also

@@ -19,64 +19,82 @@ the Hermes agent in-process from the shared `HERMES_HOME`, so it is the *same*
 agent — same memory, skills, and sessions — as the [[Hermes]] gateway and the
 [[Hermes-Dashboard]].

-**Host:** Dankclaw — **Port:** `0.0.0.0:8787` (all interfaces) — **Ingress:** SSH tunnel for the phone app; direct LAN access also works
+**Host:** Dankclaw — **Port:** `0.0.0.0:8787` (all interfaces; `ss -ltnp`
+2026-10-18 shows the webui listening there in the host netns) — **Ingress:** SSH
+tunnel for the phone app; direct LAN access also works (re-measured 2026-10-18
+from another LAN host: `/health` → 200, `/` → 302) — **Vhost:**
+`hermes.franksdank.cloud`, **exempt by construction** — no Anubis, no Authentik;
+`HERMES_WEBUI_PASSWORD` is the only gate

 ## Facts

 | Item | Value |
 |------|-------|
-| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
-| Command | `python3 /webui/bootstrap.py --foreground` |
-| Source checkout | `/home/dank/Containers/hermes/webui` on Dankclaw, bind-mounted at `/webui` |
-| Port | `8787`, bound to `0.0.0.0` in the **host** network namespace (`Network=host`) — LAN-reachable, not loopback-only |
-| Auth | `HERMES_WEBUI_PASSWORD` (host `.env`; Hermex signs in with it) |
+| Image | `localhost/hermes-agent:latest` with `Pull=never` (`dankclaw/hermes/quadlet/hermes-webui.container:35-36`) — the same image as the gateway and dashboard. The deployer keeps that alias fresh by tagging the upstream name into local storage (`podman tag docker.io/nousresearch/hermes-agent:latest localhost/hermes-agent:latest`, `scripts/deploy-remote.sh:1056`); the installed image is the 2026-09-24 build, revision `749220e` (`podman image inspect` 2026-10-18). |
+| Command | `python3 /webui/bootstrap.py --foreground` (`hermes-webui.container:48`) |
+| Source checkout | `/home/dank/Containers/hermes/webui` on Dankclaw, bind-mounted at `/webui` (`hermes-webui.container:47`; live dir `drwxrwxr-x dank dank`) |
+| Port | `8787`, bound to `0.0.0.0` in the **host** network namespace (`Network=host` + `HERMES_WEBUI_HOST=0.0.0.0`, `hermes-webui.container:37-40`) — LAN-reachable, not loopback-only |
+| Auth | `HERMES_WEBUI_PASSWORD` (from `/home/dank/serverAppConfigs/dankclaw/.env`, the unit's only `EnvironmentFile`; Hermex signs in with it) |
 | Data | bind mount `/home/dank/Containers/hermes/data` → `/data`, shared with the gateway and dashboard; anonymous volume at `/opt/data` |
-| Health | `HealthCmd=` in the Quadlet unit probes `http://127.0.0.1:8787/health` from inside the unit (works because the webui runs in the host netns); the deployer probes the same path from the host |
+| Health | `HealthCmd=` in the Quadlet unit probes `http://127.0.0.1:8787/health` from inside the unit (`hermes-webui.container:53`; works because the webui runs in the host netns). Container state 2026-10-18: **healthy**. There is no host-side probe in the deployer — the contract is the seam `tests/hermes_webui.bats`, which reads the live unit and pins the Caddyfile upstream |

 ## Why a container, not a host process

 The gateway's state files are owned by the container-mapped uid (`109999`,
-from rootless Podman's `1 → 100000/65536` subuid range). A fresh rootless
+from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` →
+`dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 =
+109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless
 namespace on Dankclaw maps identically (verified with `podman unshare
 cat /proc/self/uid_map`), so a second container using the same image reads
 and writes the shared volume with exactly the ownership the gateway already
 uses. A host-side process running as `dank` (uid 1000) could not read the
-gateway's `0600` state files.
+gateway's `0600` state files (a non-privileged read of the data dir is
+`EACCES`, checked 2026-10-18).

 ## How it works

 - `bootstrap.py` creates `/webui/.venv` on first boot, pip-installs the WebUI
  and agent requirements, then **execs** `server.py` as one foreground process
-  (`--foreground`). systemd-free: Podman supervises it via
-  `restart: unless-stopped`.
- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off; a profile
+  (`--foreground`). systemd-free inside the container: supervision is the unit's
+  `Restart=always` / `RestartSec=5` (`hermes-webui.container:60-61`) — the
+  compose-era `restart: unless-stopped` is retired with `dankclaw/compose.yaml`.
+- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off (`:45`; live
+  env confirms it, `podman inspect hermes-webui` 2026-10-18); a profile
  already exists in `/data` from the gateway.
- The healthcheck `start_period` is 600 s because the very first boot spends
+- The healthcheck `start_period` is 600 s
+  (`HealthStartPeriod=600s`, `:57`) because the very first boot spends
  minutes pip-installing into a fresh venv.

 ## Remote access (Hermex on iOS — persistent SSH tunnel)

-Only port `22` is forwarded to Dankclaw at the router, so the phone reaches the
+Only port `22` is forwarded to Dankclaw at the router *(unverified 2026-10-18:
+router configuration is not readable from this host; `sshd -T` confirms sshd
+itself listens on port 22)*, so the phone reaches the
 webui through an SSH **local forward** — no extra router ports, no third-party
 relay (no Tailscale, no Cloudflare). Implemented 2026-08-31. The tunnel is an
 *access path* for the phone, **not** a containment boundary: the server binds
 `0.0.0.0:8787`, so any host that can route to `192.168.0.143` can reach the API
 directly and needs only `HERMES_WEBUI_PASSWORD` to get in (verified 2026-09-02:
-an unauthenticated `GET /health` from another LAN machine returned HTTP 200).
+an unauthenticated `GET /health` from another LAN machine returned HTTP 200;
+re-measured from a different LAN host on 2026-10-18: `/health` → 200, `/` → 302).
 See [[Container Network Namespaces]] for why the webui is reachable this way and
 the sibling dashboard on `9119` is not.

- Dedicated `tunnel` account on Dankclaw (`/usr/sbin/nologin` shell, password
-  locked). Authentication is key-only; the account can do nothing except
+- Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994,
+  shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only
+  *(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
+  without `sudo`)*; the account can do nothing except
  authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel
  while the transport stays up for port forwarding.
- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` scopes
-  `AllowTcpForwarding yes` to that user only (plus `PermitTTY no`,
-  `X11Forwarding no`).
+- sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31)
+  scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`,
+  `X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are
+  present under `Match User tunnel` (`cat` 2026-10-18).
 - Key options on the authorized key pin the blast radius:
  `command="/usr/sbin/nologin",no-pty,restrict,port-forwarding,permitopen="localhost:8787"`
  — a leaked tunnel key can only forward to `localhost:8787`, never act as a
-  general SOCKS/SSH proxy.
+  general SOCKS/SSH proxy *(unverified 2026-10-18: the authorized_keys file
+  needs `sudo` to read)*.
 - On the phone (client side): create a key on-device (private key never
  leaves the device), hand the **public** key to the admin who appends it to
  `/home/tunnel/.ssh/authorized_keys`, then connect from the SSH client:
@@ -104,10 +122,16 @@ graph LR
 ./scripts/deploy.sh dankclaw --service hermes-webui
 ```

-`deploy.sh` copies `dankclaw/hermes/quadlet/*.container` into the quadlet unit
-directory and restarts `hermes.service`, `hermes-dashboard.service` and
+`deploy_hermes` (`scripts/deploy-remote.sh:1034-1071`; any of
+`--service hermes|hermes-dashboard|hermes-webui` routes to it,
+`scripts/deploy-remote.sh:1594`) copies `chrome-devtools-protocol.container`
+plus the three trio units into the quadlet unit directory, starts the CDP daemon
+only if inactive (it is never restarted on deploy, so in-flight browser
+sessions survive), re-tags the image alias, then restarts
+`hermes.service`, `hermes-dashboard.service` and
 `hermes-webui.service` in the **user** manager of `dank` (linger must be on for
-them to come up at boot), so gateway/dashboard/webui restarts stay isolated from
+them to come up at boot — `loginctl show-user dank -p Linger` → `yes`,
+2026-10-18), so gateway/dashboard/webui restarts stay isolated from
 inference deploys.
 Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
 `dankclaw/.env`.
@@ -118,21 +142,35 @@ Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
  the repo root. Do not flip it to `:ro`.
 - The webui binds `0.0.0.0:8787`, and **two consumers depend on that**: the SSH
  tunnel (which dials the host's `localhost`) *and* Caddy, which fronts
-  `hermes.franksdank.cloud` with `reverse_proxy 192.168.0.143:8787` — a host
-  address rather than a container name, because this unit is host-networked and
-  there is no port forwarder to bridge the gap (`dankclaw/Caddyfile` records the
-  same constraint). So the SSH tunnel is an access path for the phone, not a
+  `hermes.franksdank.cloud` with **`reverse_proxy host.containers.internal:8787`**
+  (`dankclaw/Caddyfile:186-188`) — netavark's host address (169.254.1.2), not
+  the host's LAN IP. That distinction is load-bearing: this unit is
+  host-networked and publishes nothing, so netavark DNATs nothing, and
+  `192.168.0.143:8787` is REFUSED from Caddy's bridge netns while
+  `host.containers.internal:8787` answers (measured inside the Caddy container
+  2026-09-16, recorded at `dankclaw/Caddyfile:178-185` and pinned by the seam
+  `tests/hermes_webui.bats:52-64`, which fails any Caddyfile that reverts to the
+  LAN IP). The dashboard's `9119` answers on the LAN IP precisely because it IS
+  published. So the SSH tunnel is an access path for the phone, not a
  security boundary: any LAN host that can route to `192.168.0.143` reaches the
  API and needs only `HERMES_WEBUI_PASSWORD`. The webui can run agent commands,
  so treat that password as a credential with command-execution blast radius.
+  The vhost itself carries **no** guard chain — no Anubis challenge, no
+  Authentik gate (measured 2026-10-18 from another LAN host:
+  `https://hermes.franksdank.cloud/` → 302 → `/login?next=/` and `/health` → 200,
+  with no authentik or bot-challenge bytes in the body); it is private only
+  because no public DNS record exists for the name
+  (`dankclaw/pihole/addn-hosts:39`, LAN-only).
  To make loopback-only binding true again you must set
  `HERMES_WEBUI_HOST=127.0.0.1` **and** give Caddy another way in (a
  host-gateway route or a bridge-network unit) — that is an exposure change to
  decide deliberately, not a silent fix. It was tried on 2026-09-16 as a silent
  fix and broke the vhost with a 502 while the whole e2e suite stayed green;
-  `tests/e2e/specs/17-hermes-vhost.spec.ts` now asserts that vhost reaches its
-  app through the guard chain, so a dead upstream fails a gate instead of
-  hiding.
+  `tests/e2e/specs/17-hermes-vhost.spec.ts` now asserts the vhost reaches its
+  app (no 5xx, no Caddy error page, no gate page), so a dead upstream fails a
+  gate instead of hiding. Its file comment still names the
+  `192.168.0.143:8787` upstream and calls this "the only guarded vhost" — both
+  are stale against `dankclaw/Caddyfile:186-188`.
 - The webui and the gateway share one data root (`/data` is a bind mount of
  `/home/dank/Containers/hermes/data`). A config change made on one side takes
  effect when the other restarts (same caveat as the dashboard).
@@ -12,13 +12,33 @@ timestamp: 2026-09-15
 Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.

 **Host:** Dankclaw — the gateway itself publishes **no port** (it runs in the
-host network namespace and dials the Signal daemon at `127.0.0.1:8080`); the
-user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
+host network namespace; `podman inspect systemd-hermes` → `net=host`, no port
+bindings, 2026-10-18). Note the **container name drift**: the unit is
+`hermes.service` but the generated start line is
+`podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
+`podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
+`ContainerName=` key even though its header comment says "container name:
+hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
 `hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
-record) — **Docs:** https://hermes-agent.nousresearch.com/docs
+record; the name resolves from the Pi-hole hosts file,
+`dankclaw/pihole/addn-hosts:39`) — **Docs:**
+https://hermes-agent.nousresearch.com/docs
+
+**Signal is not carrying traffic today (measured 2026-10-18).** The gateway
+boots and stays up, but its own log says
+`Platform 'signal' is explicitly disabled by platforms.signal.enabled: false in
+config.yaml, so the credentials found in the environment (SIGNAL_HTTP_URL) will
+NOT start its adapter` (`journalctl --user -u hermes` 2026-10-18), and the host
+daemon it would dial is not running either: `systemctl --user
+is-enabled signal-cli-daemon` → `disabled`, `is-active` → `inactive`, and
+nothing listens on `:8080` (`ss -ltn`). Everything below still describes the
+designed wiring; the remedy for both halves is in the log line
+(`platforms.signal.enabled: true`) plus enabling the daemon unit.

 **Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
-at `/data` in all three Hermes containers (gateway, webui, dashboard). Each
+at `/data` in all three Hermes containers (gateway, webui, dashboard —
+`hermes.container:36`, `hermes-webui.container:46`,
+`hermes-dashboard.container:24`). Each
 container additionally gets an anonymous volume at `/opt/data`, which is
 re-created empty on every container recreation — anything that lives only there
 is not durable. The tracked, sanitized configuration baseline is
@@ -28,8 +48,8 @@ is not durable. The tracked, sanitized configuration baseline is

 | Item | Value |
 |------|-------|
-| Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
-| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. |
+| Image | `localhost/hermes-agent:latest`, built from `NousResearch/hermes-agent` (official Dockerfile in repo); `Pull=never` in all three units (`hermes.container:28-29`, `hermes-webui.container:35-36`, `hermes-dashboard.container:19-20`), so the tag never moves at runtime. Live image: built 2026-09-24, OCI revision label `749220ef0007f8d87bd1531f1c24b0fe93816385` (`podman image inspect localhost/hermes-agent:latest` 2026-10-18) — that build is what is running. |
+| Upstream latest | agent v0.21.4 / CalVer `v2026.9.21` (2026-09-21); hermes-webui v0.52.113 (2026-08-14). Pin the CalVer tag, not `:latest`. *(unverified 2026-10-18: the local image records only a git revision, and the webui version endpoint answers 401 without a session, so neither number can be re-derived from the host.)* |
 | License | MIT |
 | Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
 | Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
@@ -40,16 +60,35 @@ is not durable. The tracked, sanitized configuration baseline is

 - Point Hermes at the local model endpoint or an external provider through its provider config.
 - The gateway runs `hermes gateway run --force -v` in the **host** network
-  namespace (`Network=host` in its Quadlet unit) and reaches the Signal daemon at
-  `http://127.0.0.1:8080`. That loopback hop works only because the gateway is in
+  namespace (`Exec=` + `Network=host`, `dankclaw/hermes/quadlet/hermes.container:27,37`) and reaches the Signal daemon at
+  `http://127.0.0.1:8080` (`SIGNAL_HTTP_URL`, `hermes.container:33`). That loopback hop works only because the gateway is in
  the host netns — see [[Container Network Namespaces]]. The daemon itself is a
-  **host** user unit (`signal-cli-daemon.service`), not a container, and binds
-  `0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
+  **host** user unit (`~/.config/systemd/user/signal-cli-daemon.service`), not a
+  container. **Today it is `disabled` + `inactive` and nothing is bound to
+  `:8080`** (`systemctl --user is-enabled/is-active`, `ss -ltn`, both
+  2026-10-18), and the gateway's own config gate (`platforms.signal.enabled:
+  false`) keeps the adapter off even if the daemon came back — see the dated
+  note at the top of this page. When the daemon does run it binds
+  `0.0.0.0:8080`, i.e. the Signal HTTP API is reachable from the whole LAN.
 - The data volume persists skills, memory, and session history.
- Hermes has two named local providers: `dankclaw-large` reaches
-  `http://192.168.0.143:7770/v1` and exposes `frank-dragoon`;
-  `dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
-  `frank-medium` (renamed from `frank-small` on 2026-09-15).
+- Hermes has two named local providers (the `dankclaw-large` name is confirmed
+  live from the gateway log — `resolve_provider_client: named custom provider
+  'dankclaw-large' has no resolvable api_key`, `journalctl --user -u hermes`
+  2026-10-18; the base URLs below are the repo record, since
+  `/data/config.yaml` is not readable without `sudo`):
+  `dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
+  `frank-dragoon` — verified from the host with `curl
+  http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len
+  200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is
+  a rootless pasta publish and is IPv4-only);
+  `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
+  2026-10-18, that endpoint serves only `frank-archon`** (`GET
+  http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
+  this page used to carry for it — `frank-medium`, renamed from `frank-small` on
+  2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
+  if the live `config.yaml` still names `frank-medium` the provider is dead
+  weight and requests will 404 on a nonexistent model. Diff the live config
+  before trusting either name.
 - The live `config.yaml` contains secrets and user state and is not committed;
  the tracked example is the sanitized backup of its non-secret settings.

@@ -63,9 +102,9 @@ is not durable. The tracked, sanitized configuration baseline is

 ## Gotchas

- Hermes is under active development; pin the image tag for reproducibility.
+- Hermes is under active development; pin the image tag for reproducibility. The units already pin by absence of pulling (`Pull=never`) and by building `localhost/hermes-agent:latest` locally — a rebuild, not a pull, is what changes the code.
 - Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
+- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth. The gateway itself publishes no port today; the surfaces that are exposed are the webui (`hermes.` vhost, exempt by construction because the app enforces its own password — see [[Hermes-WebUI]]) and the dashboard (`hermesadmin.` vhost → `192.168.0.143:9119`, app-enforced basic auth — see [[Hermes-Dashboard]]).

 ## Browser automation (Chrome DevTools Protocol daemon)

@@ -73,11 +112,19 @@ The agent's browser tool is an MCP client (`mcp_tool.py` in the image) that
 spawns `npx -y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`
 per session; the browser itself lives in a dedicated daemon container, the
 digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
-network, loopback-only bind on 127.0.0.1:9222). Both consumers — the Signal
-gateway and the webui server — run in the host network namespace, so they can
+network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
+`127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
+`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`,
+`dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
+gateway and the webui server — run in the host network namespace (`podman
+inspect` → `net=host` for both, 2026-10-18), so they can
 dial the daemon on loopback; nothing on the LAN can reach the CDP endpoint.
+`GET http://127.0.0.1:9222/json/version` answers **200** today with
+`Browser: Chrome/151.0.7922.109`, `Protocol-Version: 1.3` (2026-10-18).
 The wiring lives in the live `/data/config.yaml` under `mcp_servers.chrome-devtools`
-(command `npx`, args `-y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`).
+(command `npx`, args `-y chrome-devtools-mcp@latest --browserUrl=http://127.0.0.1:9222`) —
+that file is mode `drwx------` uid 109999, so reading it needs `sudo` (a
+non-privileged read is `EACCES`, checked 2026-10-18).
 The npx package cache lives in the container layer, so a recreation (not just a
 restart) pays one first-call npm download; subsequent calls are cached.

@@ -115,12 +162,15 @@ checks miss this class entirely.
 ## Live config edits

 Live file: `/home/dank/Containers/hermes/data/config.yaml` (dir `drwx------`
-uid 109999; all reads/edits need `sudo`). The repo file
+uid 109999 — `ls -ld /home/dank/Containers/hermes/data` → `drwx------ 29
+109999 109999` 2026-10-18; a non-privileged read returns `EACCES`, so all
+reads/edits need `sudo`). The repo file
 `dankclaw/hermes/config.yaml.example` is a sanitized backup and **can lag
 live — always diff the live file first; never treat the example as truth**.

 1. **Backup first:** `sudo cp -a …/config.yaml …/config.yaml.bak-YYYYMMDD`
-   (in-dir `.AGENTTRASH/` also exists for removals).
+   (in-dir `.AGENTTRASH/` also exists for removals *(unverified 2026-10-18:
+   inside the `0700` data dir, unreadable without `sudo`)*).
 2. **Apply** with surgical `sudo sed -i` or full-file write; YAML comments in
   the live file are tool-generated — preserve them.
 3. **Sweep stale references** across the whole data dir:
@@ -196,6 +246,12 @@ What actually breaks, and the fix:
          effort: none                 # mandatory, see below
  ```

+  Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw
+  `:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider
+  name is recorded against — answers only `frank-archon`. The provider/model
+  pairing above is the shape that was landed in 2026; re-derive it from the live
+  config before reusing it.
+
 - **Disable thinking for any cap-limited task.** The adapter omits reasoning
  only when config says `reasoning.enabled: false`. The
  `reasoning_effort: none` *shorthand* produces exactly that and leaves the
@@ -220,10 +276,13 @@ or a chatty non-title answer. Always read the raw content, not the status.

 ## Webui password rotation ("Hermex")

-Auth contract (verified live 2026-08-31): `POST /api/auth/login` with JSON
+Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth/login` with JSON
 `{"password":"…"}` → 200 + signed HttpOnly `hermes_session` cookie (24 h TTL)
 on match, 401 on miss. No username field. `/health` answers 200
-unauthenticated and `/` answers 302 — **never verify auth via those**; probe
+unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
+`{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
+%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via
+those**; probe
 login with a deliberately wrong password (expect 401) and the real one
 (expect 200). Details on the webui surface: [[Hermes-WebUI]].

@@ -233,17 +292,26 @@ login with a deliberately wrong password (expect 401) and the real one
   `-w %{http_code}` inline.
 2. Update BOTH env files with
   `sed -i "s/^HERMES_WEBUI_PASSWORD=.*/HERMES_WEBUI_PASSWORD=<new>/"`:
-   - `/home/dank/serverAppConfigs/.env` ([Service] EnvironmentFile)
-   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one**: the
-     quadlet's `[Container] EnvironmentFile=` is what the container reads at
-     create; rotating only the root file leaves the running container serving
-     the old secret.
+   - `/home/dank/serverAppConfigs/.env` — the repo-root checkout's env file
+     (exists, `-rw------- dank dank`, 2026-10-18) but **not** loaded by this
+     unit: keep it in sync only so other consumers do not drift.
+   - `/home/dank/serverAppConfigs/dankclaw/.env` — **the effective one, for
+     both `[Container]` and `[Service]`**: `systemctl --user show hermes-webui
+     -p EnvironmentFiles` reports exactly
+     `/home/dank/serverAppConfigs/dankclaw/.env` (2026-10-18;
+     `dankclaw/hermes/quadlet/hermes-webui.container:38,62`). That is what the
+     container reads at create; rotating only the root file leaves the running
+     container serving the old secret.
 3. `systemctl --user restart hermes-webui.service` (unit recreation re-reads
   env; no deploy needed if the unit file itself didn't change).
 4. Verify wrong-pw → 401, right-pw → 200, from host loopback **and** a LAN
   client at `http://192.168.0.143:8787` (webui publishes on the LAN
-   interface since commit `4443897`; off-network ingress is SSH tunnel only —
-   user `tunnel`, port **22**, key-only auth, forward remote end
+   interface since commit `4443897`, 2026-08-31 "publish webui on LAN interface
+   instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
+   off-network ingress is SSH tunnel only —
+   user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
+   *(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*,
+   forward remote end
   localhost:8787).

 Gotchas: `sudo podman exec` fails exit 125 here — root's namespace does not
@@ -17,8 +17,10 @@ timestamp: 2026-09-12
 Marreta is a self-hosted **content-cleaning proxy**: you hand it a URL and it
 returns a stripped, readable version of the page — no paywall chrome, no
 tracking, no cookie walls. It is a Laravel 13 app on the
-`serversideup/php:8.4-fpm-nginx` image, with an optional headless-browser
-companion for JavaScript-heavy pages.
+`serversideup/php:8.4-fpm-nginx` image (`composer.json` requires
+`php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` —
+<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>),
+with an optional headless-browser companion for JavaScript-heavy pages.

 **Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained)
 — **Upstream:** host-published port `8082` (inside the Gluetun netns)
@@ -28,11 +30,16 @@ companion for JavaScript-heavy pages.

 - Fetches a target URL with a cascade of strategies: plain HTTP (curl), a
  headless browser (Lightpanda) for JS pages, and the Internet Archive
-  (Wayback) as a last resort.
+  (Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php`
+  files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`,
+  `WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`).
 - Strips paywall/consent/tracking markup and re-renders clean HTML.
 - Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` /
-  `ADMIN_PASSWORD`) for settings and usage.
- Health endpoint at `/up` (Laravel's built-in health route).
+  `ADMIN_PASSWORD`) for settings and usage — live 2026-10-18
+  `curl http://127.0.0.1:8082/admin/login` → **200**.
+- Health endpoint at `/up` (Laravel's built-in health route —
+  `bootstrap/app.php` passes `health: '/up'`). Live: `curl
+  http://127.0.0.1:8082/up` → **200** (2026-10-18).

 ## Topology

@@ -58,11 +65,17 @@ graph LR

 | Unit | File | Image | Netns | Listens |
 |------|------|-------|-------|---------|
-| `marreta.service` | `dankclaw/marreta/quadlet/marreta.container` | `ghcr.io/manualdousuario/marreta:latest` | `gluetun.container` | `8085` (nginx) |
-| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` | `gluetun.container` | `9222` (CDP) |
+| `marreta.service` | `dankclaw/marreta/quadlet/marreta.container` | `ghcr.io/manualdousuario/marreta:latest` (`:35`) | `gluetun.container` (`:34`) | `8085` (nginx, `NGINX_HTTP_PORT=8085`, `:37`) |
+| `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) |

 Both units are boot-persistent (`WantedBy=default.target`) and self-healing
-(`Restart=always`), per the repo's Boot Persistence rule.
+(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`,
+`marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live
+2026-10-18 both report `systemctl --user is-enabled` → `generated` and
+`is-active` → `active`, and `podman inspect` reports marreta **healthy**
+(`HealthCmd` GETs `http://127.0.0.1:8085/up`, `marreta.container:53`) and
+marreta_browser `running`. Both images are `:latest`, so a redeploy can move
+them.

 ## The port-collision resolution (why 8085)

@@ -81,8 +94,11 @@ The fix is a single env var, verified against the serversideup source:
  port off the collision with no custom build.

 Marreta then publishes **nothing** of its own — the netns owner (Gluetun)
-publishes `8082:8085` on its behalf, exactly as it publishes `8081:8080`
-for SearXNG. Caddy dials `192.168.0.143:8082`.
+publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`),
+exactly as it publishes `8081:8080` for SearXNG. Caddy dials
+`192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the
+relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085`
+(2026-10-18).

 ## Why netns sharing, not `PROXY_URL`

@@ -93,7 +109,13 @@ browser ran outside the tunnel it would leak the host's real egress IP on
 every JS-rendered page. Netns sharing is the only way to put **both** the
 app and the browser on the same exit, which is what "share the same Gluetun
 egress as SearXNG" requires. The browser is therefore reached on loopback
-(`ws://127.0.0.1:9222`) inside the shared netns.
+(`ws://127.0.0.1:9222`) inside the shared netns — upstream's own compose uses
+`ws://marreta_browser:9222`, which needs no service name here because both
+containers share one namespace. Live env: `BROWSER_WS_ENDPOINT=ws://127.0.0.1:9222`
+(`podman inspect marreta` 2026-10-18; `marreta.container:44`). Upstream agrees
+on the mechanism: `BrowserFetchStrategy.php` only calls
+`BrowserFactory::connectToBrowser($wsEndpoint)` and never passes a proxy, while
+`CurlFetchStrategy.php:24-25` is the only place `marreta.proxy_url` is read.

 ## Guard chain (behind Authentik)

@@ -101,16 +123,19 @@ egress as SearXNG" requires. The browser is therefore reached on loopback
 [[Anubis]] and [[Authentik]]:

 1. **Public edge** — the vhost proxies everything to `anubis:8923`
-   (bot challenge), with `header_up X-Real-Ip {remote_host}`.
+   (bot challenge), with `header_up X-Real-Ip {remote_host}`
+   (`dankclaw/Caddyfile:159-163`).
 2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies
-   whole-request to `authentik-server:9000`.
+   whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`).
 3. **Post-auth routing** — authenticated traffic returns to `http://:8017`,
   where `@marreta host marreta.franksdank.cloud` routes to
-   `reverse_proxy 192.168.0.143:8082`.
+   `reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`).

 **Access model: any authenticated user.** The post-auth handler carries no
 group gate (unlike the homepage `/admin`, which requires the
-`homepage-admins` group). Anyone who can log in via Authentik can use
+`homepage-admins` group) — the Caddyfile says so in the block header
+("Any authentik-authenticated user may use it (no group gate)",
+`dankclaw/Caddyfile:156-157`). Anyone who can log in via Authentik can use
 Marreta.

 **Required provider (manual, one-time).** Before the vhost goes live, a
@@ -136,8 +161,12 @@ must change together.
 ## Scheme / proxy trust

 Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0.1')`
-and forces `https` when `APP_URL` starts with `https://`. With
-`APP_URL=https://marreta.franksdank.cloud` the app honors the forwarded
+— that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and
+this container sets no `TRUSTED_PROXIES`, so the default is what runs
+(`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL`
+starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`,
+`URL::forceScheme('https')`). With
+`APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded
 scheme and client IP through the whole chain, so there is no redirect loop
 or mixed-content breakage.

@@ -145,19 +174,24 @@ or mixed-content breakage.

 Three values, seeded into `/home/dank/Containers/marreta/env` (mode 0600,
 dank-owned so the rootless unit can read it — the grafana convention, not
-`/opt`):
+`/opt`; live 2026-10-18 `-rw------- dank dank` 155 B). The seeder writes it
+only when the file does not exist and `chmod 600`s it
+(`seed_marreta_env`, `scripts/deploy-remote.sh:615-622`).

 | Repo `.env` key | Mapped to (app) | Notes |
 |-----------------|-----------------|-------|
-| `MARRETA_APP_KEY` | `APP_KEY` | Laravel encryption key. **Must be the `base64:`-prefixed form**: `base64:$(openssl rand -base64 32)`. A bare base64 string (no prefix) is read as a raw key of the wrong length and every app route 500s with `Unsupported cipher or incorrect key length` at `Encrypter.php:61` (only `/up`, which skips the encrypter, still passes). Changing it after first boot invalidates encrypted session/cookie data. |
+| `MARRETA_APP_KEY` | `APP_KEY` | Laravel encryption key. **Must be the `base64:`-prefixed form**: `base64:$(openssl rand -base64 32)`. A bare base64 string (no prefix) is read as a raw key of the wrong length and every app route 500s with `Unsupported cipher or incorrect key length` at `Encrypter.php:61` (laravel/framework 13.x, `src/Illuminate/Encryption/Encrypter.php:61`) while `/up`, which skips the encrypter, still passes. Today's key is well-formed: `/`, `/admin/login` and `/up` all answer 200 (`curl http://127.0.0.1:8082/…` 2026-10-18). Changing it after first boot invalidates encrypted session/cookie data. |
 | `MARRETA_ADMIN_EMAIL` | `ADMIN_EMAIL` | Admin panel login (created on first boot). |
 | `MARRETA_ADMIN_PASSWORD` | `ADMIN_PASSWORD` | Admin panel password. |

 The quadlet maps them with `Environment=APP_KEY=${MARRETA_APP_KEY}` etc.
-The `.container` files hold **no** secrets — only `${VAR}` placeholders.
+(`dankclaw/marreta/quadlet/marreta.container:47-49`), loaded from the env file
+via `[Service] EnvironmentFile=` (`:60`). The `.container` files hold **no**
+secrets — only `${VAR}` placeholders.
 The SQLite DB + cache live in the `marreta_storage` named volume at
-`/var/www/html/storage/app` (podman auto-creates it and copyups the image's
-`www-data` ownership so the app can write).
+`/var/www/html/storage/app` (`marreta.container:52`; podman auto-creates it and
+copyups the image's `www-data` ownership so the app can write). Live 2026-10-18
+the volume exists at `/home/dank/.local/share/containers/storage/volumes/marreta_storage/_data`.

 ## Deploy

@@ -165,11 +199,11 @@ The SQLite DB + cache live in the `marreta_storage` named volume at
 ./scripts/deploy.sh dankclaw --service marreta
 ```

-`deploy_marreta` seeds the env file (only if absent), installs both
+`deploy_marreta` (`scripts/deploy-remote.sh:630-640`) seeds the env file (only if absent), installs both
 `.container` units, ensures Gluetun is up, then restarts the browser before
 the app. Because Marreta shares Gluetun's netns, the planner treats it as
 part of the Gluetun netns family: a change to `gluetun.container` or
-`searxng.network` also selects `marreta` (a Gluetun recreation tears the
+`searxng.network` also selects `marreta` (`scripts/deploy-plan.py:295-309` — a Gluetun recreation tears the
 netns down and would otherwise orphan it).

 > **First deploy note:** adding `PublishPort=8082:8085` to `gluetun.container`
@@ -180,14 +214,20 @@ netns down and would otherwise orphan it).

 - **Unauthenticated:** `curl -s https://marreta.franksdank.cloud/` → the
  Anubis challenge page (grep body for `challenge`/`not a bot`). A 502 here
-  means the chain is mis-wired.
+  means the chain is mis-wired. Re-checked 2026-10-18: **200** with `challenge`
+  ×5 and `not a bot` ×2 in the body — the guard chain is live.
 - **Post-auth routing:** `podman exec caddy sh -c "grep -n -A3 '@marreta host' /etc/caddy/Caddyfile"`
-  after the deploy's reload line.
+  after the deploy's reload line (repo copy: `dankclaw/Caddyfile:419-422`).
 - **Egress:** from inside the netns, `podman exec marreta curl -s https://ifconfig.co`
-  must return the VPN exit IP, not the home IP.
+  must return the VPN exit IP, not the home IP. The tunnel's own egress is
+  readable without exec: `journalctl --user -u gluetun` logged exit
+  `149.22.88.19` (United States, Washington, Seattle) on 2026-10-18.
 - **Port:** `podman exec marreta sh -c 'curl -fsS localhost:8085/up'` → 200;
-  `curl http://192.168.0.143:8082/up` → 200.
- **Test seam:** `bats tests/marreta_quadlet.bats` (17 contracts).
+  `curl http://192.168.0.143:8082/up` → 200 (also **200** from the host
+  loopback, `curl http://127.0.0.1:8082/up` 2026-10-18; the container health
+  probe GETs `http://127.0.0.1:8085/up` and reports **healthy**).
+- **Test seam:** `bats tests/marreta_quadlet.bats` (17 contracts — 17 `@test`
+  blocks, counted 2026-10-18).

 ### Live verification (deployed 2026-09-12, rev `91e9410`)

@@ -9,20 +9,35 @@ timestamp: 2026-09-03

 # Postgres

-PostgreSQL is the database backend for Gitea. It runs on the same podman network as Gitea and holds the repository metadata, users, and issue data.
+PostgreSQL is the database backend for two separate services on this host:
+**Gitea** (`gitea-postgres`) and **Authentik** (`authentik-postgres`). They are two
+independent clusters — two containers, two data dirs, two podman networks — and each
+holds only its own app's data; neither is a shared instance. Gitea's DB holds the
+repository metadata, users, and issue data; Authentik's holds the IdP objects.

 **Host:** Dankclaw — **Docs:** https://www.postgresql.org/docs

 ## Facts

-| Item | Value |
-|------|-------|
-| Image | `docker.io/library/postgres:16-alpine` |
-| Port | 5432 (internal only) |
-| Data volume | `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` (live bind, verified from the unit) |
+| Item | `gitea-postgres` | `authentik-postgres` |
+|------|------------------|----------------------|
+| Image | `docker.io/library/postgres:16-alpine` (`dankclaw/gitea/quadlet/gitea-postgres.container:18`) | `docker.io/library/postgres:16-alpine` (`dankclaw/authentik/quadlet/authentik-postgres.container:17`) |
+| Port | 5432, container-internal — no `PublishPort`; a TCP probe of `192.168.0.143:5432` is refused (`ss -ltn` + probe 2026-10-18) | same: nothing published, 5432 refused on the LAN IP (2026-10-18) |
+| Network | `systemd-gitea`, 10.89.0.0/24 (`dankclaw/gitea/quadlet/gitea.network`, `podman network inspect systemd-gitea` 2026-10-18) | `dankclaw_default`, 10.89.1.0/24 (`authentik-postgres.container:18`; the compose-stack bridge shared with Authentik/Seerr/Grafana) |
+| Data volume | `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` (`gitea-postgres.container:23`) | `/home/dank/Containers/authentik/postgres:/var/lib/postgresql/data` (`authentik-postgres.container:23`) |
+| Healthcheck | **none** — the unit carries no `HealthCmd`, so `podman ps` reports `Up` with no health state (2026-10-18) | `pg_isready -d authentik -U authentik` every 30 s, 5 retries, 20 s start (`authentik-postgres.container:29-33`); container reports `healthy` (2026-10-18) |
+| Secret source | `/home/dank/Containers/gitea/env` (mode 0600, `ls -l` 2026-10-18) | `/home/dank/serverAppConfigs/dankclaw/.env` (`authentik-postgres.container:19`) |
+
+`shared/homepage/services.json` reflects both: its `Postgres` card reads "Authentik's
+and Gitea's databases (Dankclaw). Not probeable: 5432 is container-internal."

 ## Configuration

+Retired compose-era definition (kept for the history below — the live stack is
+Quadlet, and this service ran from `compose.yaml` before the 2026 boot-persistence
+cutover). Note the data path in it (`/opt/gitea/data/gitea-postgres`) no longer
+exists; nothing deploys this block today.
+
 ```yaml
 services:
  postgres:
@@ -38,7 +53,9 @@ services:

 ## Gitea Connection

-Gitea reaches the database over the `systemd-gitea` network (quadlet-managed live, compose-managed after the full-stack deploy):
+Gitea reaches its database over the `systemd-gitea` network. The live quadlet unit
+`dankclaw/gitea/quadlet/gitea-server.container:24-29` sets exactly these variables
+(`Environment=DATABASE_*`), so the compose-era block below is equivalent in keys:

 ```yaml
 environment:
@@ -49,19 +66,49 @@ environment:
  DATABASE_PASSWD: "${GITEA_DB_PASSWORD}"
 ```

-The live `app.ini` on the host overrides env vars. When you rename the database container, update its `[database] HOST` line. See [[Gitea]] Gotchas.
+The live `app.ini` on the host overrides env vars: its `[database]` stanza carries
+`HOST = gitea-postgres:5432`, `NAME = gitea`, `USER = gitea`
+(`/home/dank/Containers/gitea/data/gitea/conf/app.ini:27-33`, 2026-10-18). When you
+rename the database container, update that `[database] HOST` line. See [[Gitea]]
+Gotchas.

 ## Layout (current)

-The live `gitea-postgres` unit mounts its data at
-`/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data` — the old
+Both live units mount their data under `/home/dank/Containers`:
+
+- `gitea-postgres` → `/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data`
+- `authentik-postgres` → `/home/dank/Containers/authentik/postgres:/var/lib/postgresql/data`
+
+Both directories exist on the host with mode 0700 and owner uid 100069 (the mapped
+`postgres` uid of the rootless namespace) — `ls -ld` 2026-10-18. The old
 `/opt/gitea/data/gitea-postgres` compose-era path is gone. Secrets come from
-the host-local env file `/home/dank/Containers/gitea/env` (0600) via
-`EnvironmentFile=`; `POSTGRES_PASSWORD` never enters this repo. See [[Gitea]].
+host-local env files (`/home/dank/Containers/gitea/env`, 0600;
+`/home/dank/serverAppConfigs/dankclaw/.env`) via `EnvironmentFile=`; the passwords
+never enter this repo. See [[Gitea]] and [[Authentik]].
+
+## Boot persistence (Quadlet)
+
+Both units carry `Restart=always` (`gitea-postgres.container:28`,
+`authentik-postgres.container:38`) and `[Install] WantedBy=default.target`
+(`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed
+under `~/.config/containers/systemd/`, and report `generated` / `active` with
+`Linger=yes` on the host (2026-10-18), so both come back at boot.

 ## Healthcheck

-Use `pg_isready`:
+Only `authentik-postgres` has one. Its quadlet runs bare argv, not a shell string,
+because the generator space-splits `HealthCmd` (the quoted `/bin/sh -c "…"` form
+arrived mangled and reported a false-negative `unhealthy` forever — see the comment
+in `dankclaw/authentik/quadlet/authentik-postgres.container:24-33`):
+
+```ini
+HealthCmd=pg_isready -d authentik -U authentik
+HealthInterval=30s
+```
+
+`gitea-postgres` has **no** `HealthCmd`; the probe below is the compose-era snippet,
+kept as the recipe to add if Gitea's DB ever needs one. Until it is in the unit,
+judge `gitea-postgres` by `podman ps` state and Gitea's own health.

 ```yaml
 healthcheck:
@@ -73,7 +120,12 @@ healthcheck:

 ## Gotchas

- The database password must match between the `postgres` service and Gitea's `DATABASE_PASSWD`; both come from the same `.env` variable.
+- Two clusters, not one: `gitea-postgres` serves only Gitea and
+  `authentik-postgres` only Authentik. They are on different bridges
+  (`systemd-gitea` 10.89.0.0/24 vs `dankclaw_default` 10.89.1.0/24, `podman network
+  inspect` 2026-10-18), so neither can reach the other's data by name. Any
+  "one Postgres serves everything" reading of this page is wrong.
+- The database password must match between the `postgres` container and Gitea's `DATABASE_PASSWD`; both come from the same env-file variable (`GITEA_DB_PASSWORD`, expanded by systemd from `/home/dank/Containers/gitea/env`).
 - Do not change the Gitea `SECRET_KEY` or `INTERNAL_TOKEN` after migration — encrypted data becomes unreadable.
 - Back up with `pg_dump`; see [[Gitea]] (knowledge base).

@@ -81,3 +133,10 @@ healthcheck:

 - [[Gitea]]
 - [[Dankclaw Stack]]
+
+## Citations
+
+- `dankclaw/gitea/quadlet/gitea-postgres.container`, `dankclaw/gitea/quadlet/gitea.network`, `dankclaw/gitea/quadlet/gitea-server.container`
+- `dankclaw/authentik/quadlet/authentik-postgres.container`
+- `shared/homepage/services.json` (the `Postgres` card)
+- Host evidence 2026-10-18: `podman ps`, `podman inspect`, `podman network inspect systemd-gitea dankclaw_default`, `ss -ltn`, TCP probe of `192.168.0.143:5432`, `systemctl --user is-enabled/is-active gitea-postgres authentik-postgres`
@@ -14,12 +14,15 @@ VPN tunnel, and how to unblock them in SearXNG.

 **Host:** Dankclaw — **Docs:** https://docs.searxng.org

-SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
-Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
+SearXNG aggregates the engines bundled in the running build — **262 installed, 75
+enabled** on the live instance (`GET http://127.0.0.1:8081/config`, 2026-10-18; the
+"~50 engines" framing this page carried predates that count). When all of them are
+queried through **one** VPN exit IP — as on
 Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
 start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
-engine for an hour to 15 days and silently drops it from results. That is the
-"Google (and sometimes a few others) went unavailable" symptom.
+engine (three minutes for a 429 under the current build's defaults, up to 15 days for
+a Cloudflare CAPTCHA — see the table below) and silently drops it from results. That
+is the "Google (and sometimes a few others) went unavailable" symptom.

 This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
 are the detail if you need it.
@@ -35,20 +38,32 @@ should disappear completely.
 |----------|------------------|----------------------|
 | General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
 | Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
-| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
+| Science | arXiv, PubMed, Semantic Scholar | Google Scholar until independently tested; OpenAlex (`openalex` is `enabled: false` in the running build, `/config` 2026-10-18) |
 | News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
-| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
+| Images | Wikimedia Commons (ships as `wikicommons.images/videos/audio/files`, all enabled), Bing Images, Unsplash | Openverse (`enabled: false` today), Google/Brave/Startpage image engines |
 | Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |

-This policy uses no API credentials and does not identify the operator to a
-search provider. It is deliberately conservative: a bundled engine is not
+**Live state of this table (2026-10-18, from `GET /config`):** the General/web,
+Technical, News and Videos baselines hold engine-for-engine; the two rows above are
+the corrections. Nothing in the deferred column is enabled. One operational note for
+the deferred engines: re-enabling Google CSE on demand needs a key this host does not
+hold — `/opt/searxng/.env` carries only the WireGuard variables plus
+`SEARXNG_SECRET`, `SEARXNG_BASE_URL` and `KAGI_API_KEY` (key **names** read
+2026-10-18; values not printed).
+
+This policy was written to use no API credentials and to avoid identifying the
+operator to a search provider — **with one carve-out made afterwards (2026-09-26):
+the paid `kagi` engine, whose key is rendered into the deployed config from
+`/opt/searxng/.env` (`shared/searxng/settings.yml:70`; item 4 below and
+[[SearXNG]]).** It is deliberately conservative: a bundled engine is not
 necessarily healthy, anonymous, or suitable for every default query.

 ## Root cause (one line)

 All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
-Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
-rate-limit it → CAPTCHA → engine **suspension** → empty results.
+Google/DuckDuckGo see one datacenter IP spamming every enabled engine (75 of the 262
+installed ones, `/config` 2026-10-18) and rate-limit it → CAPTCHA → engine
+**suspension** → empty results.

 ## The suspension model (what "unavailable" actually is)

@@ -58,15 +73,23 @@ Suspended engines are the usual cause of "gone":

 | Exception | Default suspension | Trigger |
 |-----------|--------------------|---------|
-| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
-| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
-| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
-| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
-| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
-| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
+| `SearxEngineTooManyRequests` | **180 s (3 min)** | HTTP 429 |
+| `SearxEngineAccessDenied` | **180 s (3 min)** | 403 / access denied |
+| `SearxEngineCaptcha` | **3600 s (1 h)** | Google `/sorry/` or reCAPTCHA |
+| `recaptcha_SearxEngineCaptcha` | ~7 days (604 800 s) | Google reCAPTCHA |
+| `cf_SearxEngineCaptcha` | ~15 days (1 296 000 s) | Cloudflare edge CAPTCHA |
+| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail` 5 / `max_ban_time_on_fail` 120 |

-The values are `search.suspended_times` (+ `ban_time_on_fail`) in
-`settings.yml`. An engine that returns **0 results without an error** is *not*
+These are the defaults of the **running commit** `12f8b6515`, read from
+[`searx/settings.yml` at that commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml)
+(lines 66-81). The folklore numbers this table used to carry — ~1 hour for 429, ~1 day
+for access-denied and CAPTCHA — were the **older** upstream defaults and are retired;
+the current build un-suspends a 429'd engine after three minutes.
+
+The values are `search.suspended_times` (+ `ban_time_on_fail`). Our
+`shared/searxng/settings.yml` overrides **none** of them (`grep -n "suspended_times\|ban_time_on_fail"`
+returns nothing, 2026-10-18), so the upstream defaults quoted above are exactly what
+runs. An engine that returns **0 results without an error** is *not*
 suspended — that is the separate "selector drift" problem (below).

 ## Two failure modes
@@ -81,9 +104,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
 ## Fix A — Engine config (settings.yml `engines:`)

 The current tracked policy enables Bing and DuckDuckGo for default general
-search, gives each weight 2, disables pagination, and allows 5 seconds for the
-VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
-but disabled from default aggregation. This limits fan-out while preserving
+search, gives each weight 2, disables pagination, and allows 5.0 seconds for the
+VPN path (`shared/searxng/settings.yml:26,38-56`). CAPTCHA/429-prone Google, Brave,
+and Startpage families are retained but disabled from default aggregation
+(`:81-124`) — though the three `startpage*` stanzas are dead weight: the running
+build ships no `startpage` engine at all (`/config` lists none, 2026-10-18), so those
+overrides match nothing. This limits fan-out while preserving
 explicit/on-demand selection.

 Important: SearXNG's engines are configured in the **`engines:` list inside
@@ -105,7 +131,7 @@ engines:
  - name: google
    weight: 3
    disabled: false
-    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
+    timeout: 5.0                 # wait longer over the tunnel (upstream global default is 3.0s — searx/settings.yml:178 at the running commit)
    retry_on_http_error: [403, 429]
    display_error_messages: true

@@ -172,8 +198,8 @@ Tune the `outgoing` block and the suspend model to survive load:

 ```yaml
 outgoing:
-  request_timeout: 4.0             # current Dankclaw value; modest VPN-latency allowance
-  max_request_timeout: 12.0
+  request_timeout: 5.0             # Dankclaw's value (shared/searxng/settings.yml:26)
+  max_request_timeout: 15.0        # Dankclaw's cap (shared/searxng/settings.yml:27)
  pool_connections: 100
  pool_maxsize: 20
  enable_http2: true
@@ -194,9 +220,9 @@ Notes:
  that is a historical-searx concept. The functional equivalent is the suspend
  schedule above (widen to back off before blocking, narrow to recover faster).
 - `request_timeout` interacts with the suspend model: too tight → engines look
-  "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
-  12 seconds as the cap; raise it only when measurements show real upstream
-  latency rather than 429/CAPTCHA responses.
+  "unresponsive"; too loose → slow pages. Dankclaw uses 5 seconds globally and
+  15 seconds as the cap (`shared/searxng/settings.yml:26-27`); raise it only when
+  measurements show real upstream latency rather than 429/CAPTCHA responses.
 - Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
  provider's rate limit worse. A retry count of 1 is a bounded compromise for a
  transient network failure.
@@ -281,7 +307,14 @@ The read-only audit found:
  Google's "unusual traffic from your network" response.
 - `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
  healthy. This isolates the primary failures to upstream provider handling,
-  not application availability.
+  not application availability. Re-checked 2026-10-18: `/healthz` is still `200`
+  and `gluetun`/`searxng`/`searxng-valkey` are all `generated` + `active`.
+
+What has moved since that audit: the container is still
+`ghcr.io/searxng/searxng:latest` (`podman inspect searxng` 2026-10-18), but the
+embedded version is now `2026.9.25+12f8b6515` (`GET /config`), not `1.0.0`, and
+`limiter.enabled` is `true`. The exit IP was not re-measured (that would need a
+query fired through the tunnel).

 This deployment returns the legacy array form for `unresponsive_engines`:

@@ -293,10 +326,15 @@ This deployment returns the legacy array form for `unresponsive_engines`:
 ```

 Do not assume the newer object form (`engine`, `error_type`, `suspended`)
-until the live response has been checked. The direct published port is the
-useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
-hostname resolved to the WAN address during the audit and timed out from the
-audit host, so it was not used as the baseline path.
+until the live response has been checked — and note that the jq recipes below key on
+`.engine`, which only matches the object form; against the legacy `[engine, message]`
+pair they silently match nothing. *(Re-checked 2026-10-18 with a `bing`-only JSON
+query: `unresponsive_engines` came back empty, so today's shape is unverified.)* The
+direct published port is the useful diagnostic path because it exercises the
+ProtonVPN exit. The Caddy hostname resolved to the WAN address during the audit and
+timed out from the audit host, so it was not used as the baseline path; on
+2026-10-18 `search.franksdank.cloud` resolves to the LAN IP from inside the network
+and answers with the Anubis challenge page.

 ## Fix D — Verify an engine is unblocked

@@ -304,7 +342,9 @@ Three layers, cheapest first. All queries should hit the path production uses
 (the **published Gluetun port → ProtonVPN exit**, not a direct host port).

 **1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
-(otherwise `format=json` → 403), then query a single engine:
+(otherwise `format=json` → 403) — `html` and `json` are both enabled here
+(`shared/searxng/settings.yml:18-20`), and a JSON query through the published port
+returns 200 `application/json` today (2026-10-18). Query a single engine:

 ```bash
 BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
@@ -335,7 +375,10 @@ done
 ```

 **2. `/config` REST endpoint** lists every engine + enabled/disabled +
-categories — an inventory of what is live. `?engines=<name>` isolates a single
+categories — an inventory of what is live. It answers 200 on this instance today
+(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
+`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
+[[SearXNG]]). `?engines=<name>` isolates a single
 provider; `/debug` adds engine diagnostics.

 **3. `check.searx.space`** — the hosted official checker probes every engine of
@@ -369,14 +412,17 @@ misleading).
   engine (Kagi) is acceptable when the blast radius is bounded — the
   `kagi` engine runs with `paging: false` and weight 2, so it spends exactly
   one API credit per aggregated query, and its key never enters git
-   (deploy-time rendering from `/opt/searxng/.env`; see [[SearXNG]]). It is
+   (deploy-time rendering from `/opt/searxng/.env` into
+   `/opt/searxng/config/settings.yml`, where the `${KAGI_API_KEY}` placeholder is
+   already replaced — checked 2026-10-18; see [[SearXNG]]). It is
   also immune to the shared-VPN-exit-IP blocking this page is about: it
   authenticates, so reputation of the exit IP is irrelevant.
 5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
   exactly which engines are down and why. Confirm the running SearXNG version
   (check whether `parsers`/`rate_limit` exist in the pinned build).
 6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
-   upstream request, caps it at 15 seconds, and retries once. This addresses
+   upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
+   (`:27`), and retries once (`:32`). This addresses
   modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
   after measuring the resulting error and latency profile.
 7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
@@ -407,6 +453,8 @@ misleading).
 - [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
 - [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
 - [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
+- [Upstream `searx/settings.yml` at the running commit `12f8b6515`](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml) — the `suspended_times`, `ban_time_on_fail` and `request_timeout` defaults quoted above.
+- [Upstream `searx/engines/kagi.py` at the same commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/engines/kagi.py) — Bearer auth, `POST https://kagi.com/api/v1/search`, `require_api_key: True`.

 ## See Also

@@ -17,11 +17,11 @@ SearXNG is a privacy-respecting metasearch engine. It aggregates results from ma

 | Item | Value |
 |------|-------|
-| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
-| Port | 8081 (host, on Gluetun) → 8080 (container) |
-| Config | `/etc/searxng/settings.yml` (bind mount) |
-| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
-| Formats | `html`, `json` (agent API) |
+| Image | `ghcr.io/searxng/searxng:latest` (`dankclaw/searxng/quadlet/searxng.container:19`; the live container's image reference is exactly that — `podman inspect searxng` 2026-10-18). Running build: `2026.9.25+12f8b6515` (`GET /config`) |
+| Port | SearXNG listens on **8080 inside Gluetun's network namespace**; the host sees **8081**, published by the gluetun unit (`dankclaw/searxng/quadlet/gluetun.container:31`; `ss -ltn` shows `*:8081`, `curl http://127.0.0.1:8081/` → 200, 2026-10-18) |
+| Config | `/etc/searxng/settings.yml` (bind mount of `/opt/searxng/config`, `searxng.container:22`) |
+| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine`, container `searxng-valkey`, publishes nothing (see [[Valkey]]) |
+| Formats | `html`, `json` (agent API) — `shared/searxng/settings.yml:18-20`; `curl "http://127.0.0.1:8081/search?q=…&format=json"` → 200 `application/json` (2026-10-18) |

 ## VPN Routing (critical)

@@ -37,15 +37,15 @@ Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]]

 ## Boot Persistence (Quadlet)

-The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
+The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `dankclaw/searxng/quadlet/` and install to `~/.config/containers/systemd/`.

 The units live in `dankclaw/searxng/quadlet/` and install to
 `~/.config/containers/systemd/` on Dankclaw (verified live):

 | Unit file | Service | Role |
 |-----------|---------|------|
-| `searxng.network` | generated network service | DNS-enabled bridge; the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 |
-| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` — the only published port in the trio |
+| `searxng.network` | generated network service | DNS-enabled bridge (`dns_enabled: true`, `podman network inspect systemd-searxng` 2026-10-18); the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 (`searxng.network:16`) |
+| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` for SearXNG **and** `8082:8085` for [[Marreta]] (`gluetun.container:31-32`) — SearXNG's is the only port in the search trio, but the tunnel unit carries two, and both are live (`ss -ltn` 2026-10-18) |
 | `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache on the searxng network, alias `valkey` (see [[Valkey]]) |
 | `searxng.container` | `searxng.service` | `Network=gluetun.container` — joins the tunnel's netns, so it publishes no port of its own; reached through gluetun's 8081 |

@@ -53,7 +53,7 @@ gluetun and valkey join the searxng network. searxng reuses gluetun's netns
 and reaches valkey by name on that network. Each unit sets
 `WantedBy=default.target`. User linger is on, so the units start at boot.

-Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
+Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw` — a wrapper that execs `scripts/deploy.sh dankclaw --service searxng` (`scripts/deploy-searxng-quadlet.sh:11`). The dispatcher copies the config, installs the three units plus the network, daemon-reloads, and starts them in dependency order: network → gluetun → valkey → searxng (`scripts/deploy-remote.sh:583-593`).

 ## settings.yml Keys

@@ -108,51 +108,66 @@ engines:
    disabled: true
 ```

+The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
+the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
+`secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
+censused below.
+
 ## Volumes

-| Container path | Host path (convention) |
-|----------------|------------------------|
-| `/etc/searxng` | `/opt/searxng/config` |
-| `/etc/searxng/valkey` | (valkey container data) |
+| Container path | Host path | Proof |
+|----------------|-----------|-------|
+| `/etc/searxng` (searxng) | `/opt/searxng/config` | `searxng.container:22`; live dir holds `settings.yml` + `limiter.toml` (`ls -la /opt/searxng/config` 2026-10-18) |
+| `/data` (valkey container) | `/opt/searxng/valkey` | `dankclaw/searxng/quadlet/searxng-valkey.container:20` — this is a mount of the **valkey** container, not of `/etc/searxng/valkey` |

 ## Healthcheck

-SearXNG responds on `/healthz` with 200 when ready.
+SearXNG responds on `/healthz` with 200 when ready — re-checked on 2026-10-18:
+`curl -s -o /dev/null -w '%{http_code} %{content_type}' http://127.0.0.1:8081/healthz`
+→ `200 text/plain`. The unit itself carries no `HealthCmd`, so `podman ps` shows
+`searxng` as plain `Up` (2026-10-18); the homepage card probes `127.0.0.1:8081`.

 ## Trusted Group (limiter.toml)

-The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
+The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on (`server.limiter: true`, `shared/searxng/settings.yml:11` — and the running instance confirms it: `GET /config` reports `limiter.enabled: true`, 2026-10-18). The file deploys to `/opt/searxng/config/limiter.toml` (`scripts/deploy-remote.sh:211`), mounted at `/etc/searxng/limiter.toml`. The deployed copy is byte-identical to the repo file today (`diff` clean, 2026-10-18).

 Two lists control access:

 - `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
 - `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).

-The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
+The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`) — `shared/searxng/limiter.toml:54-60`. An IP in `pass_ip` skips every bot-detection check, including the rate limiter: ip_lists "outranks every other method" (`limiter.toml:41-45`). `deploy.sh` copies the file into place on each deploy (`scripts/deploy-remote.sh:208-211`).

 Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.

 The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.

 Reach SearXNG through Caddy (`search.franksdank.cloud`, guarded) for browser
-use. A local subagent can hit the published port directly because its
+use: the vhost challenges through `anubis:8923` (`dankclaw/Caddyfile:102-103`) and the
+post-auth listener :8017 routes it to `192.168.0.143:8081` (`dankclaw/Caddyfile:361`).
+A local subagent can hit the published port directly because its
 slirp/loopback/LAN sources are in `pass_ip`. The searxng network
-(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`, so Caddy's
-`X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it
-stays stable across reboots.
+(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`
+(`shared/searxng/limiter.toml:22-29`), so Caddy's `X-Forwarded-For` is honored. The
+subnet is pinned in `searxng.network:16`, so it stays stable across reboots — that is
+what keeps the `trusted_proxies` entry meaningful.

 ## Unblocking Google & Search Providers

-The current request policy allows a modest amount of VPN latency (`4.0s` per
-request, `12.0s` maximum) while limiting retries to one. This can recover slow
+The current request policy allows a modest amount of VPN latency (`5.0s` per
+request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
+retries to one (`:32`). This can recover slow
 providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
-VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
-IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
-The default provider policy therefore favors anonymous Bing and DuckDuckGo,
-disables observed CAPTCHA/429 providers from normal aggregation, and disables
-pagination to reduce upstream request volume. SearXNG **suspends** offending
-engines for hours to days, which is what shows up as "Google went unavailable".
-This is upstream, not local.
+VPN exit IP. SearXNG queries every engine it has enabled from one Gluetun VPN exit
+IP: the running build ships 262 engines and **75 are enabled**
+(`curl http://127.0.0.1:8081/config` → `engines[]`, 2026-10-18). The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
+The default provider policy therefore favors anonymous Bing and DuckDuckGo (both `enabled: true`
+in `/config`), disables observed CAPTCHA/429 providers from normal aggregation
+(google, google cse, google news, google scholar, the whole `brave.*` family:
+`enabled: false` on 2026-10-18), and disables pagination for those engines to reduce
+upstream request volume. SearXNG **suspends** offending engines — which is what shows
+up as "Google went unavailable". The default suspension is shorter than the folklore:
+see the table in [[SearXNG Providers]]. This is upstream, not local.

 Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
 `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
@@ -165,10 +180,11 @@ root cause, the suspend-model table, the `engines:` config, and the
 ### Gotchas

 - The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
- **Agent-API contract (verified live 2026-08-31):** the guarded path
-  (`search.franksdank.cloud`) answers non-browser API clients with Anubis'
+- **Agent-API contract (verified live 2026-08-31, re-verified 2026-10-18):** the guarded
+  path (`search.franksdank.cloud`) answers non-browser API clients with Anubis'
  JS-challenge page at HTTP **200** — a status-code-only client silently gets
-  HTML where it expected JSON. Check `content-type`, not just status. There is
+  HTML where it expected JSON (`curl` with a JSON query through the vhost returned
+  `200 text/html`, 2026-10-18). Check `content-type`, not just status. There is
  deliberately no way for an API client through the guard (JS challenge + UA
  deny-list); the agent path is the direct published port
  `http://192.168.0.143:8081/search?q=…&format=json` from trusted sources —
@@ -176,38 +192,95 @@ root cause, the suspend-model table, the `engines:` config, and the
  LAN by construction (no public route to the LAN IP). No second agent-only
  instance is needed; a duplicate would double upstream request volume from the
  same rate-sensitive VPN exit IP for zero benefit.
- The config file is root-owned (uid of the rootless podman namespace). The
-  official path is `deploy.sh --service searxng`: it copies the repo settings
+- The config dir is owned by uid 100976 — the searxng container's mapped root inside
+  the rootless user namespace, not a host root account
+  (`ls -la /opt/searxng/config` 2026-10-18: both files `100976 100976`).
+  The official path is `deploy.sh --service searxng`: it copies the repo settings
  and renders the `${VAR}` placeholders via `scripts/searxng-render-settings.py`
-  from `/opt/searxng/.env`. A hotfix can still hand-copy with `sudo cp`, but
-  remember the renderer: an unrendered copy silently serves the literal
+  from `/opt/searxng/.env` (`scripts/deploy-remote.sh:208-216`). A hotfix can still hand-copy with `sudo cp`, but remember the renderer: an unrendered copy silently serves the literal
  placeholders (before 2026-09-26 the live `secret_key` literally was
-  `"${SEARXNG_SECRET}"` and nobody noticed).
- `base_url` must match the public address or proxy rewrites break.
- Keep the valkey cache container on the same podman network as searxng.
+  `"${SEARXNG_SECRET}"` and nobody noticed). Today's deployed file carries no
+  unrendered placeholder — the only `${` left in it is inside the header comment
+  (`grep -n '[$]{' /opt/searxng/config/settings.yml`, 2026-10-18).
+- `base_url` must match the public address or proxy rewrites break — **the deployed
+  value does not**: `SEARXNG_BASE_URL` in `/opt/searxng/.env` is set so that the
+  rendered config reads `base_url: "http://192.168.0.143:8081/"`
+  (`/opt/searxng/config/settings.yml:10`, 2026-10-18), i.e. the published port on the
+  LAN IP, not `https://search.franksdank.cloud`. Browser use through the vhost still
+  works because Caddy rewrites `Host`, but absolute links SearXNG builds point at the
+  LAN address. The unit also passes `SEARXNG_BASE_URL` into the container
+  (`searxng.container:21`), and that is one of the variables SearXNG substitutes at
+  load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
+  config change, out of scope for docs.
+- Keep the valkey cache container on the network SearXNG can actually reach — see
+  [[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
+  netns), so the network that matters is the one Gluetun and `searxng-valkey` share.


 ## Kagi (paid API engine)

-Added 2026-09-26 (user-ordered). The stock SearXNG image ships a native
-`kagi` engine module (`searx/engines/kagi.py`); it authenticates with
-`Authorization: Bearer <key>` against `POST https://kagi.com/api/v1/search`.
+Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
+`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
+which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
+`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
+and declares `require_api_key: True` (`kagi.py:63`).

 - **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
-  `[general, web]`, shortcut `kg`, `kagi_categ: search`.
+  `[general, web]`, shortcut `kg`, `kagi_categ: search`
+  (`shared/searxng/settings.yml:66-77`); live `/config` reports `kagi` with
+  `enabled: true`, categories `general, web` (2026-10-18).
 - **Cost control**: `paging: false` — exactly one API credit per aggregated
-  query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0).
+  query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
+  (`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
+  `kagi.py:67`, so the `false` here is a real override).
 - **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
  `deploy-remote.sh` bakes the real key into the live settings.yml via
  `scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
-  (dank-owned, mode 600). The key never enters git and never appears in any
-  process argv. SearXNG itself does NOT expand env vars in settings.yml —
-  without the renderer the engine fails `setup()` and is dropped at load.
+  (owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
+  the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
+  The key never enters git and never appears in any
+  process argv. SearXNG does not expand *arbitrary* `${VAR}` placeholders in
+  settings.yml — it substitutes only a fixed set of documented environment variables
+  (`SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `SEARXNG_VALKEY_URL`, …; see the header
+  comments in upstream `searx/settings.yml` at the pinned commit), and `${KAGI_API_KEY}`
+  is not one of them. Without the renderer the engine fails `setup()` and is dropped
+  at load (`kagi.py:91` returns the error when the key is empty). The deployed file
+  has no unrendered placeholder left (2026-10-18).
 - **Immune to the VPN-exit-IP problem**: Kagi authenticates, so it does not
  care about the shared Gluetun exit IP reputation (see [[SearXNG Providers]]).
 - **Verified live 2026-09-26**: direct API probe through the gluetun netns
  (200, `data.search[]` shape), then a live aggregated query returned
-  29/29 results tagged `kagi`, zero unresponsive engines.
+  29/29 results tagged `kagi`, zero unresponsive engines. Re-checked 2026-10-18 only
+  as far as the config goes (engine enabled, key rendered); no paid query was fired
+  for this audit.
+
+## Live engine census (2026-10-18)
+
+`GET http://127.0.0.1:8081/config` answers `engines[]` for the running instance
+(200 today; the endpoint would be blocked on a public instance, and
+`public_instance: false` here). Counted from that response:
+
+- 262 engines in the build, **75 enabled**, categories of the enabled set led by
+  `other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
+- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
+  `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
+  `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
+  `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
+  `youtube`, `vimeo`, `dailymotion`, `bing videos`.
+- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
+  `brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
+- The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
+  engines that **no longer exist in this build** — `/config` lists no `startpage`,
+  `startpage images` or `startpage news`, so those overrides are no-ops kept as
+  intent. *(labelled 2026-10-18)*
+
+## Citations
+
+- `dankclaw/searxng/quadlet/searxng.container`, `gluetun.container`, `searxng-valkey.container`, `searxng.network`
+- `shared/searxng/settings.yml`, `shared/searxng/limiter.toml`
+- `scripts/deploy-searxng-quadlet.sh:11`, `scripts/deploy-remote.sh:208-216,583-593`
+- `dankclaw/Caddyfile:102-103,361` (guarded vhost → post-auth → `192.168.0.143:8081`)
+- Host evidence 2026-10-18: `podman ps`, `podman inspect searxng`, `podman network inspect systemd-searxng`, `ss -ltn`, `curl 127.0.0.1:8081/` → 200, `/healthz` → 200, `/config` → 200 (engine census), `/search?format=json` → 200 `application/json`, `diff shared/searxng/limiter.toml /opt/searxng/config/limiter.toml` (identical), `systemctl --user is-enabled/is-active gluetun searxng searxng-valkey`

 ## See Also

@@ -9,26 +9,28 @@ timestamp: 2026-09-02

 # Seerr

-Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line). Frank's household uses it to request movies/TV/music; Seerr routes approvals to Sonarr/Radarr/Lidarr/Readarr on FrankBook-Pro and pushes downloads to qBittorrent.
+Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line). It is meant to route approvals to Sonarr/Radarr/Lidarr/Readarr on FrankBook-Pro and push downloads to qBittorrent. **It is not launched:** `shared/homepage/services.json` ships the card with `wip: "down"`, `showLink: false` and the description "Work in progress: not open to family users yet", and `/srv/homepage/status.json` (2026-10-18) still reports it `wip-down` even though the container is up. Treat every "Frank's household uses it" framing as future tense.

 **Host:** Dankclaw — **Hostname:** `seerr.franksdank.cloud` (Caddy, guarded) — **Docs:** https://github.com/seerr-team/seerr

-## Facts (verified live, 2026-09-01)
+## Facts (verified live 2026-09-01, re-verified 2026-10-18)

 | Item | Value |
 |------|-------|
-| Image | `ghcr.io/seerr-team/seerr:v3.4.1` (runs as in-image `node`, uid 1000) |
-| Port | 5055 (container), published `127.0.0.1:5055` on the host (loopback-only) |
-| Data | `/home/dank/Containers/seerr/data` → `/app/config` (SQLite settings store) |
-| Unit | `dankclaw/seerr/quadlet/seerr.container` → `seerr.service` |
-| Network | `dankclaw_default` (post-auth listener reachability) |
-| Guard | Caddy TLS edge → Anubis :8923 → Authentik outpost provider `seerr-proxy` → Caddy :8017 `@seerr` handler → `seerr:5055` |
-| DNS | Pi-hole addn-hosts only. The public Porkbun A record was deleted on 2026-09-02, so the name resolves on the LAN (→ 192.168.0.143) and returns NXDOMAIN on the WAN: Seerr is internal-only by construction |
+| Image | `ghcr.io/seerr-team/seerr:v3.4.1` (unit pin `dankclaw/seerr/quadlet/seerr.container:32`; live container reference `podman inspect seerr` 2026-10-18). Runs as the in-image `node` user, uid 1000 (`podman top seerr user uid`) |
+| Port | 5055 (container), published `127.0.0.1:5055` only (`seerr.container:34`; `ss -ltn` shows `127.0.0.1:5055` and no `0.0.0.0` bind, 2026-10-18) |
+| Data | `/home/dank/Containers/seerr/data` → `/app/config` (SQLite settings store, `seerr.container:35`; the live dir exists and is owned by uid 100999 — see UID mapping) |
+| Unit | `dankclaw/seerr/quadlet/seerr.container` → `seerr.service`; `Restart=always`/`RestartSec=5` (`:38-39`), `[Install] WantedBy=default.target` (`:42-43`), installed under `~/.config/containers/systemd/` and reported `generated` / `active` with `Linger=yes` (2026-10-18) — boot-persistent |
+| Network | `dankclaw_default` (post-auth listener reachability); live address 10.89.1.12 on the 10.89.1.0/24 bridge (`podman inspect seerr`, `podman network inspect dankclaw_default` 2026-10-18) |
+| Guard | Caddy TLS edge → Anubis :8923 → Authentik outpost provider `seerr-proxy` *(unverified 2026-10-18: the provider name lives in Authentik's own DB)* → Caddy :8017 `@seerr` handler → `seerr:5055` (`dankclaw/Caddyfile:149-150` for the challenged vhost, `:371` for the post-auth `reverse_proxy seerr:5055`) |
+| DNS | Pi-hole addn-hosts only: `192.168.0.143 seerr.franksdank.cloud` (`dankclaw/pihole/addn-hosts:46`). The public Porkbun A record was deleted on 2026-09-02, so the name resolves on the LAN (→ 192.168.0.143) and not on the WAN — re-checked 2026-10-18: a public-resolver query for `seerr.franksdank.cloud` returns no A record, while `ai.franksdank.cloud` returns the WAN address. Seerr is internal-only by construction |

 ## Why loopback-only publishing

 Seerr is browser-facing; the only sanctioned user path is the guard chain (Caddy → Anubis → Authentik SSO). `PublishPort=127.0.0.1:5055:5055` keeps the app off the LAN entirely while still letting the homepage probe agent and host-local checks reach it — the same loopback convention as other guarded hosts (see [[Homepage Design Grilling]]).

+There is no health endpoint to point at: measured on 2026-10-18, `GET http://127.0.0.1:5055/` answers **307** (redirect into the app) and `GET /api/v1/version` answers **404** — Seerr does not expose that path, so do not wire it as a health probe. The homepage card probes `127.0.0.1:5055` and is marked `wip: "down"`, which is why the card stays hidden while the container is up.
+
 ## Guard chain fix (2026-09-02)

 The `seerr-proxy` provider was provisioned by script with **no flows and no
@@ -43,18 +45,23 @@ provisioning rules this incident established.

 ## *arr wiring

-Seerr reaches the *arr services on FrankBook-Pro (192.168.0.112, gluetun-NetNS stack inside the Mac's podman machine VM) by IP-based base URLs (`http://192.168.0.112:<port>`), configured inside Seerr's own web settings — not in env or quadlet. The `.lan` hostname variant was deliberately dropped: addn-hosts records are for fleet public-domain hosts, and IP bases match the Mac stack's verified LAN reachability. API keys live in each arr's config.xml on the Mac and are entered through Seerr's settings store.
+Seerr reaches the *arr services on FrankBook-Pro (192.168.0.112, gluetun-NetNS stack inside the Mac's podman machine VM) by IP-based base URLs (`http://192.168.0.112:<port>`), configured inside Seerr's own web settings — not in env or quadlet. The `.lan` hostname variant was deliberately dropped: `dankclaw/pihole/addn-hosts` carries no `frankbook-pro.lan` record (verified against the file, 2026-10-18), and IP bases match the Mac stack's verified LAN reachability. *(unverified 2026-10-18: 192.168.0.112 is offline at this census — `ping` fails and the arr probes report `down` in `/srv/homepage/status.json`, so LAN reachability from Seerr could not be re-measured; the quadlet header's "resolved via the Pi-hole addn-hosts record frankbook-pro.lan" is stale for the same reason — no such record exists.)* API keys live in each arr's config.xml on the Mac and are entered through Seerr's settings store. *(unverified 2026-10-18: the keys are inside Seerr's SQLite store, not readable from the repo.)*

 OIDC single-sign-on for Seerr (Authentik OIDC provider + Seerr OIDC config) remains an open option; the guard chain already authenticates the front door, so this is an in-app setting deferred to Frank's first-run session.

 ## UID mapping gotcha

-The container runs as `node` (uid 1000 in-image). Rootless podman's continuous mapping is container uid N → host uid `100000 + N - 1`, so uid 1000 → **100999**. The first deploy chowned the data dir to 100999 + 1 (= 101000) and crash-looped with `EACCES mkdir /app/config/logs`. Rule: read the real mapping from any container's `/proc/self/uid_map` before chowning bind roots for non-root images.
+The container runs as `node` (uid 1000 in-image — `podman top seerr user uid` 2026-10-18). Rootless podman's mapping on this host is `0 → 1000 (size 1)`, `1 → 100000 (size 65536)` (`/proc/<pid>/uid_map` of the running container; `podman unshare cat /proc/self/uid_map` agrees), i.e. container uid N ≥ 1 → host uid `100000 + N - 1`, so uid 1000 → **100999**. That is the owner the data tree actually carries: `ls -n /home/dank/Containers/seerr/data` shows `100999 100999` on `db/`, `logs/` and the files the app writes (2026-10-18). The first deploy chowned the data dir to 100999 + 1 (= 101000) and crash-looped with `EACCES mkdir /app/config/logs`. Rule: read the real mapping from a running container's `/proc/<pid>/uid_map` before chowning bind roots for non-root images.

 ## Deployment

-`./scripts/deploy.sh dankclaw --service seerr`. The deployer pre-creates the bind-source dir, installs the unit, and restarts. First visit after deploy: complete the Seerr local-admin setup wizard (the guard chain bounces through Authentik first). TLS issuance for a new vhost fires at Caddy reload (dns-01) — allow ~15 s before the first HTTPS probe.
+`./scripts/deploy.sh dankclaw --service seerr` (`seerr` is a registered service in `scripts/deploy.sh:143`). The deployer pre-creates the bind-source dir, installs the unit, and restarts. First visit after deploy: complete the Seerr local-admin setup wizard (the guard chain bounces through Authentik first). TLS issuance for a new vhost fires at Caddy reload (dns-01) — allow ~15 s before the first HTTPS probe.

 ## Citations

 - Seerr image and volume contract: https://github.com/seerr-team/seerr (official docs recommend `/app/config` persistence)
+- Quadlet unit: `dankclaw/seerr/quadlet/seerr.container` (image pin `:32`, loopback publish `:34`, volume `:35`, `Restart=always`/`WantedBy` `:38-43`)
+- Guard chain: `dankclaw/Caddyfile:149-150` (challenged vhost) and `dankclaw/Caddyfile:371` (post-auth `reverse_proxy seerr:5055`)
+- DNS: `dankclaw/pihole/addn-hosts:46`; public-resolver query for `seerr.franksdank.cloud` (no record), 2026-10-18
+- Homepage state: `shared/homepage/services.json` (`Seerr` card, `wip: "down"`) and `/srv/homepage/status.json` 2026-10-18 (`wip-down`)
+- Host evidence 2026-10-18: `podman ps`, `podman inspect seerr`, `podman top seerr user uid`, `ss -ltn`, `curl 127.0.0.1:5055/` → 307, `curl 127.0.0.1:5055/api/v1/version` → 404, `systemctl --user is-enabled/is-active seerr`
@@ -16,15 +16,17 @@ timestamp: 2026-09-12

 | Item | Value |
 | --- | --- |
-| Swapfile | `/swap.img` |
-| Size | **64 GiB** (raised from 8 GiB on 2026-09-12) |
-| Filesystem | ext4 — swapfiles fully supported, no btrfs `nocow` handling needed |
-| Allocation | `fallocate` (preallocated, hole-free; 530 extents) |
-| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` |
-| systemd unit | `swap.img.swap` — `loaded (/etc/fstab; generated)`, `active` |
-| `vm.swappiness` | `60` (distro default, unchanged) |
-| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`) |
-| Free on `/` after | 748 GiB of 1.9 TiB |
+| Swapfile | `/swap.img` (68,719,476,736 B = 64 GiB, owner root 0600, mtime 2026-09-12 21:56 — `ls -la /swap.img`) |
+| Size | **64 GiB** (raised from 8 GiB on 2026-09-12); `/proc/swaps` reports 67,108,860 KiB |
+| Type / priority | `file`, priority **-1** — the only swap device on the host (`swapon --noheadings` 2026-10-18: `/swap.img file 64G 187.6M -1`) |
+| Filesystem | ext4 on `/dev/nvme0n1p2` (`df -T /` 2026-10-18) — swapfiles fully supported, no btrfs `nocow` handling needed |
+| Allocation | `fallocate` (preallocated, hole-free; **530 extents** — `sudo filefrag -v /swap.img` → "530 extents found", 2026-10-18) |
+| Persistence | `/etc/fstab` line 12: `/swap.img none swap sw 0 0` (`grep -n -i swap /etc/fstab` 2026-10-18) |
+| systemd unit | `swap.img.swap` — `LoadState=loaded`, `FragmentPath=/run/systemd/generator/swap.img.swap`, `UnitFileState=generated`, `is-active` → `active` (2026-10-18) |
+| `vm.swappiness` | `60` (distro default, unchanged — `sysctl vm.swappiness` 2026-10-18) |
+| zswap | disabled (`/sys/module/zswap/parameters/enabled` = `N`, 2026-10-18) |
+| Swap in use | 187 MiB of 63 GiB (`free -h` 2026-10-18) |
+| Free on `/` | **424 GiB of 1.9 TiB, 77 % used** (`df -T /` 2026-10-18). The 748 GiB that stood right after the 2026-09-12 resize is gone — ordinary fleet writes since then |

 The unit is **generated from fstab**, not a hand-written unit file. fstab is the
 only control point; `systemctl daemon-reload` re-reads it. Nothing in this
@@ -33,12 +35,19 @@ it is not invisible.

 ## Why 64 GiB

-Dankclaw is shared. `frank-dragoon` alone holds **103.5 GiB of the 121.2 GiB**
-host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
-**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
-engine itself warns that when that runs out, frank-dragoon and its neighbours
-"can stall for minutes at a time at full CPU with no disk activity and no
-output". Swap is the valve that turns an OOM kill into slow paging.
+Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
+**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
+working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
+working set was ~4.3 GiB. That is a decision-time reading, not today's: on
+2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
+(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
+`frank-dragoon-halogen` container accounts for **70.33 GiB**
+(`podman stats --no-stream`, same day) on the current flash-server payload —
+a different model and KV shape than the one measured in September.
+The engine itself warned at the time that when the headroom runs out,
+frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
+no disk activity and no output". Swap is the valve that turns an OOM kill into slow
+paging.

 Swap does **not** relieve the pinned-weights problem — see the caveat below.
 It protects the *co-tenants* and buys headroom for burst allocation.
@@ -84,10 +93,10 @@ under memory pressure.
 ## Verification

 ```bash
-swapon --show=NAME,TYPE,SIZE,USED,PRIO   # /swap.img file 64G
-systemctl is-active swap.img.swap        # active
-grep swap /etc/fstab                     # line 12 intact -> boot persistence
-free -h | tail -2                        # Swap: 63Gi total
+swapon --show=NAME,TYPE,SIZE,USED,PRIO   # /swap.img file 64G 187.6M -1 (2026-10-18)
+systemctl is-active swap.img.swap        # active; FragmentPath=/run/systemd/generator/swap.img.swap
+grep -n swap /etc/fstab                  # line 12 intact -> boot persistence
+free -h | tail -2                        # Swap: 63Gi total, 187Mi used (2026-10-18)
 ```

 Then confirm the inference service is unaffected — a completion, not just
@@ -113,9 +122,10 @@ frank-dragoon `health=200` throughout.
  pool and working set *are* anonymous. Swapping the KV pool would be far worse
  than the OOM kill it prevents. If the box ever starts thrashing, lowering
  swappiness (or capping the engine's cgroup) is the knob, not more swap.
- **`free` still lies after the change.** `MemAvailable` reads ~70 GiB while
-  the model is loaded. Trust the engine's `host memory left for everything
-  else` line, not `free`.
+- **`free` still lies after the change.** On 2026-09-12 `MemAvailable` read ~70 GiB
+  while the model was loaded; on 2026-10-18 it reads **29 GiB** with 90 GiB sitting in
+  `buff/cache` (`free -h`). The headline number moves with the payload; trust the
+  engine's `host memory left for everything else` line, not `free`.

 - **Filesystem prerequisite (other hosts).** ext4 and xfs take swapfiles
  directly. btrfs needs `chattr +C` (NOCOW), no compression, and no
@@ -131,9 +141,13 @@ frank-dragoon `health=200` throughout.

 ## Trash

-The previous 8 GiB `/swap.img` (dated 2026-07-21) is staged, not deleted, at
-`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). Purge
-only on explicit approval.
+The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
+`dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
+directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
+`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
+`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
+`20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
+The old swapfile has therefore been purged; nothing on the host still holds it.

 ## See Also

@@ -9,7 +9,7 @@ timestamp: 2026-08-28

 # Valkey

-Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment.
+Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment — the running build (2026.9.25+12f8b6515, `/config` 2026-10-18) reads a `valkey:` settings key and has no `redis:` key (`searx/settings.yml` at commit `12f8b6515`, https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml).

 **Host:** Dankclaw — **Docs:** https://valkey.io

@@ -17,13 +17,21 @@ Valkey is the cache and limiter store for SearXNG. It replaces Redis as the defa

 | Item | Value |
 |------|-------|
-| Image | `valkey/valkey:9-alpine` |
-| Port | 6379 (internal, on the SearXNG network) |
+| Image | `docker.io/valkey/valkey:9-alpine` (`dankclaw/searxng/quadlet/searxng-valkey.container:19`; live container image reference `podman inspect searxng-valkey` 2026-10-18) |
+| Port | 6379 container-internal — the unit publishes **no** port (`dankclaw/searxng/quadlet/searxng-valkey.container` has no `PublishPort`; `ss -ltn 2026-10-18` shows nothing on 6379 and a TCP probe to `192.168.0.143:6379` is refused), so `shared/homepage/services.json` marks the card `wip: "closed"` |
 | Role | SearXNG limiter + result cache |

 ## Why Valkey

-The official SearXNG docker-compose ships a `searxng-valkey` container. Valkey is the Redis fork under the Linux Foundation umbrella; the SearXNG project moved to it as the default cache.
+Valkey is the Redis fork under the Linux Foundation umbrella, and the SearXNG project
+moved to it as the default cache: the pinned build's `searx/settings.yml` exposes
+`valkey: { url: … }` and has no `redis:` key
+(https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml).
+The `searxng-valkey` service name and the `valkey/valkey:9-alpine` image are this
+repo's choice (`dankclaw/searxng/quadlet/searxng-valkey.container:16-19`), matching
+the container name SearXNG's settings then point at. *(unverified 2026-10-18: the
+claim that upstream's own docker-compose ships a container named `searxng-valkey`
+is not checkable from this repo)*

 ## Configuration

@@ -38,25 +46,38 @@ Image=docker.io/valkey/valkey:9-alpine
 Volume=/opt/searxng/valkey:/data
 ```

-SearXNG connects to it via `settings.yml`:
+SearXNG connects to it via `settings.yml`, which addresses the **container name**,
+not the alias — the alias is a second, working name for the same endpoint:

 ```yaml
 valkey:
  url: "valkey://searxng-valkey:6379/0"
 ```

-It sets `WantedBy=default.target`. User linger is on, so it starts at boot.
+Repo source `shared/searxng/settings.yml:22-23`; the deployed render
+`/opt/searxng/config/settings.yml:23` is byte-identical on this key (2026-10-18).
+
+It sets `WantedBy=default.target` and `Restart=always`
+(`dankclaw/searxng/quadlet/searxng-valkey.container:25-28`), the unit is installed at
+`~/.config/containers/systemd/searxng-valkey.container`, and `systemctl --user
+is-enabled` reports `generated` / `is-active` `active` with `Linger=yes`
+(host check 2026-10-18), so it starts at boot.

 ## Volumes

-| Container path | Host path (convention) |
-|----------------|------------------------|
-| `/data` | `/opt/searxng/valkey` |
+| Container path | Host path |
+|----------------|-----------|
+| `/data` | `/opt/searxng/valkey` (`dankclaw/searxng/quadlet/searxng-valkey.container:20`; the live dir exists, owner uid 100998 — `ls -ld /opt/searxng/valkey` 2026-10-18) |

 ## Gotchas

- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart.
- Keep Valkey on the same podman network as SearXNG.
+- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart. *(unverified 2026-10-18: upstream fallback behaviour not read; the pinned default is `url: false`, i.e. no cache configured)*
+- The cache is reachable because both containers share one network: `searxng-valkey`
+  joins `searxng.network` (`searxng-valkey.container:17`), and SearXNG itself has no
+  network of its own — it reuses Gluetun's netns (`searxng.container:18`), and Gluetun
+  is the member of `systemd-searxng` (10.89.2.0/24, `podman network inspect
+  systemd-searxng` 2026-10-18). Keeping the cache off that network breaks SearXNG's
+  limiter, not just the result cache.

 ## See Also