docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
This commit is contained in:
Frank committed
1 parent
6d4a96a423
commit
5829a0bfe8
140 files changed
+1820
-86
No files matched your search
@@ -0,0 +1,79 @@
|
||||
# ComfyUI
|
||||
|
||||
ComfyUI is a node-based AI image generation engine. Workflows are built from nodes that load models, sample latents, and post-process images; everything is scriptable through a web UI and an API.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `comfyui.dankclaw.local` — **Port:** 8188 — **Backend:** ROCm GPU (gfx1151) — **Docs:** https://github.com/Comfy-Org/ComfyUI
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui` |
|
||||
| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` |
|
||||
| Port | 8188 (host) → 8188 (container) |
|
||||
| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
|
||||
| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
|
||||
| Run | `podman compose -f dankclaw/compose.yaml up -d comfyui` |
|
||||
| Boot-persistent | user systemd unit `comfyui.service` (runs the compose `up` at boot) |
|
||||
|
||||
## Why the GPU works here
|
||||
|
||||
Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S`, capability `(11, 5)`. See [[ComfyUI ROCm and Vulkan]] for the full verification.
|
||||
|
||||
## How the image is built
|
||||
|
||||
`dankclaw/Containerfile.comfyui` starts from the AMD gfx1151 base image and bakes the ComfyUI code in. It installs ComfyUI's dependencies but skips the torch family (torch, torchvision, torchaudio). The base image already ships the matching ROCm builds. Installing an unpinned `torch` from `requirements.txt` clobbers the gfx1151 kernels with a default build, so the Containerfile filters those lines out.
|
||||
|
||||
## GPU device access (rootless)
|
||||
|
||||
The container reaches the GPU through the `/dev/kfd` and `/dev/dri` device passthrough. The permission is checked host-side. The run user is already a member of the `render` group on the host, so the device is readable.
|
||||
|
||||
Only the `video` group is added to the container. The `render` group (gid 990) is not added. Rootless podman cannot map it into the container namespace (the subgid range is 100000+), so `--group-add render` fails to start the container.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|----------------|-----------|
|
||||
| `/opt/ComfyUI/models` | `/home/dank/comfyui-data/models` |
|
||||
| `/opt/ComfyUI/input` | `/home/dank/comfyui-data/input` |
|
||||
| `/opt/ComfyUI/output` | `/home/dank/comfyui-data/output` |
|
||||
| `/opt/ComfyUI/custom_nodes` | `/home/dank/comfyui-data/custom_nodes` |
|
||||
|
||||
The ComfyUI code is baked into the image (not mounted). Rebuild the image to update the code.
|
||||
|
||||
## Custom nodes
|
||||
|
||||
Custom nodes go in `/home/dank/comfyui-data/custom_nodes` (mounted at `/opt/ComfyUI/custom_nodes`). ComfyUI loads every subdirectory there on start. Because this path is a bind mount, it shadows anything baked into the image — so custom nodes are installed host-side, not in the Containerfile.
|
||||
|
||||
**ComfyUI-Manager** is installed here. Install it with a one-time clone, then restart the container:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ltdrdata/ComfyUI-Manager \
|
||||
/home/dank/comfyui-data/custom_nodes/ComfyUI-Manager
|
||||
systemctl --user restart comfyui
|
||||
```
|
||||
|
||||
ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` (returns a version like `V3.41`). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.
|
||||
|
||||
## Boot persistence
|
||||
|
||||
The user systemd unit `~/.config/systemd/user/comfyui.service` runs `podman compose -f /home/dank/serverAppConfigs/dankclaw/compose.yaml up -d comfyui` at boot. It sets `WantedBy=default.target`, and user linger is on. The old CPU venv unit is preserved at `comfyui.service.venv.bak` as a fallback.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device).
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The GPU path needs the device passthrough. Do not remove `/dev/kfd` or `/dev/dri` from the compose service.
|
||||
- Do not add the `render` group to the container. Rootless podman cannot map it. Add only `video`.
|
||||
- Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
|
||||
- The base image is about 32 GB. A code update means a rebuild, because the code is baked in.
|
||||
- The model collection starts empty. Add models to `/home/dank/comfyui-data/models` before generating.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
|
||||
- [[Lemonade and sd.cpp]] — the ggml-family alternative
|
||||
- Knowledge base: [[ComfyUI]] (tools)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
type: reference
|
||||
title: Dankclaw Stack
|
||||
description: Service inventory and routing overview for the Dankclaw AI, development, and observability host.
|
||||
tags:
|
||||
- dankclaw
|
||||
- services
|
||||
- containers
|
||||
- caddy
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||
hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
|
||||
ROCmFPX llama.cpp), the agent tools, the privacy search, and the
|
||||
observability stack. The llama.cpp endpoint (`frank-default-small`,
|
||||
`:7770`) runs on Dankserver (192.168.0.235).
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
U[User] --> C[Caddy :80/:443]
|
||||
C --> O[OpenWebUI]
|
||||
C --> S[SearXNG]
|
||||
C --> G[Gitea]
|
||||
C --> F[ComfyUI]
|
||||
C --> D[Grafana]
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
O --> E[CIRU llama.cpp :7770]
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Hostname | Port | Docs |
|
||||
|---------|----------|------|------|
|
||||
| SearXNG | search.dankclaw.local | 8081 | [[SearXNG]] |
|
||||
| ComfyUI | comfyui.dankclaw.local | 8188 | [[ComfyUI]] |
|
||||
| OpenWebUI | chat.dankclaw.local | 3005 | [[OpenWebUI]] |
|
||||
| Hermes | hermes.dankclaw.local | 8090 | [[Hermes]] |
|
||||
| Gitea | gitea.dankclaw.local | 3000 | [[Gitea]] |
|
||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||
| Homepage | dankclaw.local | 80/443 | — |
|
||||
| frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
|
||||
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
|
||||
|
||||
The last two rows are bare-metal systemd services, not containers. This
|
||||
repo owns their scripts, units, and knowledge base (see
|
||||
[serverAppConfigs-docs/LLM/index.md](../LLM/index.md)). Current mapping (2026-08-22):
|
||||
`frank-default-large` runs the CIRU ROCmFPX llama.cpp on Dankclaw;
|
||||
`frank-default-small` runs the upstream llama.cpp on Dankserver.
|
||||
|
||||
## Boot Persistence
|
||||
|
||||
The Gitea database and server, the SearXNG trio (gluetun, searxng, valkey), open-webui, Pi-hole, and Caddy run as Quadlet units, so they start at boot. Caddy left `compose.yaml` on 2026-08-26 after a reboot exposed that bare compose containers never come back; see [[Caddy]] for the unit and the incident. The former standalone Gitea Caddy unit is retired. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent — per AGENTS.md (Boot Persistence & Self-Healing) they each need a quadlet before they may be considered reliably deployed.
|
||||
|
||||
Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`, and Pi-hole with `scripts/deploy-pihole-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]); Caddy is deployed through `scripts/deploy.sh dankclaw --service caddy`.
|
||||
|
||||
## Routing Rules
|
||||
|
||||
- SearXNG outbound traffic exits through the [[Gluetun]] VPN container.
|
||||
- Every other service uses the normal LAN route.
|
||||
- The homepage lists all services with status.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[index]] — repository index
|
||||
- [[Network Architecture]] — routing and proxy
|
||||
- [[Observability]] — Grafana, Prometheus, logs
|
||||
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
type: configuration
|
||||
title: Gitea
|
||||
description: Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service.
|
||||
tags:
|
||||
- gitea
|
||||
- dankclaw
|
||||
- quadlet
|
||||
- caddy
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# Gitea
|
||||
|
||||
Gitea is a self-hosted Git service with issues, pull requests, and a web UI. On Dankclaw it runs as two Quadlet-managed containers in the rootless Podman session of user `dank`. The canonical Caddy on the same host reverse-proxies its web UI at `gitea.franksdank.cloud`.
|
||||
|
||||
**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com
|
||||
|
||||
## Containers
|
||||
|
||||
| Container | Unit | Image | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` | Database |
|
||||
| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` | Web UI, API, git (HTTP :3000, SSH :2222) |
|
||||
|
||||
The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name. The automatic `podman` default network has DNS disabled. Do not run the Gitea containers on it.
|
||||
|
||||
## Hosted repositories
|
||||
|
||||
| Repository | Purpose |
|
||||
|------------|---------|
|
||||
| `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
|
||||
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
|
||||
|
||||
The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
|
||||
|
||||
## Management
|
||||
|
||||
Run all commands as `dank` on Dankclaw.
|
||||
|
||||
| Action | Command |
|
||||
|--------|---------|
|
||||
| Start all | `systemctl --user start gitea-postgres gitea-server` |
|
||||
| Stop all | `systemctl --user stop gitea-server gitea-postgres` |
|
||||
| Restart Gitea only | `systemctl --user restart gitea-server` |
|
||||
| Status | `systemctl --user status gitea-postgres gitea-server` |
|
||||
| Container logs | `podman logs -f gitea-server` |
|
||||
| Unit logs | `journalctl --user -u gitea-server -f` |
|
||||
|
||||
The units start in dependency order:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
N[gitea-network.service] --> P[gitea-postgres.service]
|
||||
P --> S[gitea-server.service]
|
||||
```
|
||||
|
||||
## Boot persistence
|
||||
|
||||
User-level linger is enabled for `dank` (`loginctl enable-linger dank`). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target`. The podman generator applies it at boot; no symlink is needed.
|
||||
|
||||
The containers use the quadlet fresh-start model. A stop removes the container (`podman run --rm --replace`). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.
|
||||
|
||||
## Secrets
|
||||
|
||||
The database password lives in `/opt/gitea/.env` on the host (mode 0600). The quadlet files reference it with `EnvironmentFile=`. The password never enters this repository. Placeholders live in [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example).
|
||||
|
||||
Do not change the password in place. Change the PostgreSQL role first, then update `/opt/gitea/.env` and the `[database] PASSWD` line in the app.ini (see Gotchas), then restart the stack.
|
||||
|
||||
## Configuration files
|
||||
|
||||
| File | Location | Repo copy |
|
||||
|------|----------|-----------|
|
||||
| Quadlet units | `~/.config/containers/systemd/gitea-{postgres,server}.container`, `gitea.network` | [`dankclaw/gitea/quadlet/`](../../dankclaw/gitea/quadlet/) |
|
||||
| Caddyfile | `/home/dank/Containers/caddy/Caddyfile` | [`dankclaw/Caddyfile`](../../dankclaw/Caddyfile) |
|
||||
| Secrets | `/opt/gitea/.env` (0600) | [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example) |
|
||||
|
||||
After editing a repo copy, deploy through `scripts/deploy.sh dankclaw`. The Gitea database/server units are installed from this directory; the canonical Caddy and its rendered Caddyfile are deployed as the separate `caddy` service. Do not manually recreate the retired `gitea-caddy` unit.
|
||||
|
||||
## Data volumes
|
||||
|
||||
| Container path | Host path | Contents |
|
||||
|----------------|-----------|----------|
|
||||
| `/data` | `/opt/gitea/data/gitea-gitea` | Git repositories, app.ini, attachments |
|
||||
| `/var/lib/postgresql/data` | `/opt/gitea/data/gitea-postgres` | PostgreSQL data |
|
||||
|
||||
Back up `/opt/gitea/data/` (tar) or `pg_dump` the database. See the knowledge base page [[Gitea]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **app.ini wins over env vars.** Gitea writes `/data/gitea/conf/app.ini` (host: `/opt/gitea/data/gitea-gitea/gitea/conf/app.ini`) on first start. The `[database] HOST` and `PASSWD` lines persist and override the `DATABASE_HOST`/`DATABASE_PASSWD` env vars. When you rename the database container, edit the `HOST` line.
|
||||
- **The files in the data directory are owned by host UID 100999**, not `dank` (rootless ID mapping: container uid 1000 → host 100999). Edit them inside the container with `podman exec --user 1000 gitea-server ...`.
|
||||
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
|
||||
- **Port 3000 vs 3001.** The live host port is 3000. The full-stack [`dankclaw/compose.yaml`](../../dankclaw/compose.yaml) maps 3001. Align the two before deploying the full stack.
|
||||
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
|
||||
- **Canonical proxy.** `gitea.franksdank.cloud` is served by the Caddy service in `dankclaw/compose.yaml`, which proxies to `192.168.0.143:3000`. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. During migration, the deployer stages any installed legacy unit under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy.
|
||||
|
||||
## Incident: 2026-08-16 — down after unattended reboot
|
||||
|
||||
`unattended-upgrades` rebooted Dankclaw at 15:01 on 2026-08-16. The Gitea containers ran through podman-compose with `restart: unless-stopped` and no systemd units. Rootless podman does not auto-start containers at boot without units. Gitea stayed down until a manual start at 16:02. The fix: migrate the stack to quadlet units with linger (this page). After the fix, verify on the next reboot.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Postgres]]
|
||||
- Knowledge base: [[Gitea]] (tools)
|
||||
@@ -0,0 +1,74 @@
|
||||
# Gluetun
|
||||
|
||||
Gluetun is a lightweight VPN client container. It connects to a VPN provider (WireGuard or OpenVPN) and shares that tunnel with other containers that attach to its network namespace.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://github.com/qdm12/gluetun
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `qmcgaw/gluetun:latest` |
|
||||
| Capabilities | `NET_ADMIN` required |
|
||||
| Device | `/dev/net/tun:/dev/net/tun` |
|
||||
| Ports published | searxng's 8081 → 8080 lives here |
|
||||
| Tunnel check | the `tun0` interface exists (the `:8000` server 404s on all paths) |
|
||||
|
||||
## Required Environment
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
VPN_SERVICE_PROVIDER: "${VPN_SERVICE_PROVIDER}" # e.g. mullvad, protonvpn
|
||||
VPN_TYPE: wireguard
|
||||
WIREGUARD_PRIVATE_KEY: "${WIREGUARD_PRIVATE_KEY}" # from .env, never committed
|
||||
```
|
||||
|
||||
- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it.
|
||||
- Provider-specific variables come from the Gluetun wiki; set them in `.env`.
|
||||
|
||||
## Optional DNS (from research)
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `DNS_UPSTREAM_RESOLVER_TYPE` | `dot` | DNS over TLS |
|
||||
| `DNS_UPSTREAM_RESOLVERS` | `cloudflare` | upstream DoT providers |
|
||||
| `DNS_CACHING` | `on` | cache queries |
|
||||
| `BLOCK_MALICIOUS` | `on` | block malicious hostnames |
|
||||
|
||||
Warning from the wiki: do NOT set `DNS_UPSTREAM_PLAIN_ADDRESSES` to a LAN/bridge address — DNS queries leak out of the tunnel.
|
||||
|
||||
## Attaching a Container
|
||||
|
||||
The searxng unit reuses gluetun's network namespace:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
```
|
||||
|
||||
`Network=gluetun.container` also adds a systemd dependency on gluetun, so searxng starts after the tunnel.
|
||||
|
||||
## Boot Persistence
|
||||
|
||||
gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and user linger is on, so the tunnel starts at boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600). Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`.
|
||||
|
||||
## Verify the Tunnel
|
||||
|
||||
```bash
|
||||
podman exec gluetun curl -s ifconfig.me
|
||||
```
|
||||
|
||||
The returned IP must differ from the host IP.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The container that holds the published port must be the one with the ports mapping (gluetun).
|
||||
- `restart: always` on gluetun so the tunnel reconnects after VPN drops.
|
||||
- When gluetun restarts, attached containers inherit the new network state; searxng may need a restart to re-resolve DNS.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the consumer of the tunnel
|
||||
- [[Dankclaw Stack]]
|
||||
- Knowledge base: [[SearXNG]]
|
||||
@@ -0,0 +1,41 @@
|
||||
# Hermes-Dashboard
|
||||
|
||||
The Hermes web dashboard is a browser UI for managing the agent. It shows config, API keys, and sessions. It is a management tool. It is not a chat window. Chat happens over Signal or the CLI. See [[Hermes]].
|
||||
|
||||
**Host:** Dankclaw — **Port:** `9119` — **URL:** `http://192.168.0.143:9119`
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
|
||||
| Command | `hermes dashboard --host 0.0.0.0 --port 9119 --no-open --skip-build` |
|
||||
| Port | `9119` (published to the LAN) |
|
||||
| Auth | HTTP basic auth (username + scrypt password hash) |
|
||||
| Data | shares `/home/dank/hermes-data` with the gateway |
|
||||
|
||||
## Why a separate container
|
||||
|
||||
The gateway container runs `hermes gateway run` (the Signal platform). That entrypoint does not start the dashboard. The dashboard is a separate process. So it runs in its own compose service (`hermes-dashboard`). Both services mount the same data volume. Both restart at boot (`restart: unless-stopped`).
|
||||
|
||||
## Authentication
|
||||
|
||||
The dashboard refuses to bind to a non-loopback address without an auth provider. There is no unauthenticated public bind. This repo uses basic auth:
|
||||
|
||||
- `dashboard.basic_auth.username` and `dashboard.basic_auth.password_hash` live in the live `/home/dank/hermes-data/config.yaml`; the tracked backup at `dankclaw/hermes/config.yaml.example` intentionally omits the password hash.
|
||||
- The stored value is a one-way scrypt hash. The plaintext password is never written to a file or the repo.
|
||||
- The hash is generated with `plugins.dashboard_auth.basic.hash_password`.
|
||||
|
||||
## Access
|
||||
|
||||
Open `http://192.168.0.143:9119` in a browser on the LAN. Log in with the configured username and password.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The dashboard and the gateway share one data volume. A config change made in the dashboard takes effect when the gateway restarts.
|
||||
- If the shared password is reused elsewhere, treat it as exposed. Rotate it and regenerate the hash.
|
||||
- `--skip-build` uses the prebuilt web assets in the image. Do not remove it unless the assets are rebuilt.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-WebUI]] — the phone-facing API server (Hermex)
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
type: configuration
|
||||
title: Hermes-WebUI (Hermex server)
|
||||
description: The hermes-webui compose service on Dankclaw — the loopback-only API server that the Hermex iPhone app talks to over an SSH tunnel.
|
||||
tags:
|
||||
- hermes
|
||||
- hermes-webui
|
||||
- hermex
|
||||
- dankclaw
|
||||
- ssh-tunnel
|
||||
timestamp: 2026-09-14
|
||||
---
|
||||
|
||||
# Hermes-WebUI (Hermex server)
|
||||
|
||||
`hermes-webui` is the [hermes-webui](https://github.com/nesquena/hermes-webui)
|
||||
server (MIT, third-party) that the **Hermex** iPhone app connects to. It runs
|
||||
the Hermes agent in-process from the shared `HERMES_HOME`, so it is the *same*
|
||||
agent — same memory, skills, and sessions — as the [[Hermes]] gateway and the
|
||||
[[Hermes-Dashboard]].
|
||||
|
||||
**Host:** Dankclaw — **Port:** `127.0.0.1:8787` (loopback only) — **Ingress:** SSH tunnel
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
|
||||
| Command | `python3 /webui/bootstrap.py --foreground` |
|
||||
| Source checkout | `/home/dank/hermes-webui` on Dankclaw (mounted at `/webui`) |
|
||||
| Port | `8787`, published to `127.0.0.1` only |
|
||||
| Auth | `HERMES_WEBUI_PASSWORD` (host `.env`; Hermex signs in with it) |
|
||||
| Data | shares `/home/dank/hermes-data` with the gateway and dashboard |
|
||||
| Health | `GET /health` (compose healthcheck + deployer) |
|
||||
|
||||
## Why a container, not a host process
|
||||
|
||||
The gateway's state files are owned by the container-mapped uid (`109999`,
|
||||
from rootless Podman's `1 → 100000/65536` subuid range). A fresh rootless
|
||||
namespace on Dankclaw maps identically (verified with `podman unshare
|
||||
cat /proc/self/uid_map`), so a second container using the same image reads
|
||||
and writes the shared volume with exactly the ownership the gateway already
|
||||
uses. A host-side process running as `dank` (uid 1000) could not read the
|
||||
gateway's `0600` state files.
|
||||
|
||||
## How it works
|
||||
|
||||
- `bootstrap.py` creates `/webui/.venv` on first boot, pip-installs the WebUI
|
||||
and agent requirements, then **execs** `server.py` as one foreground process
|
||||
(`--foreground`). systemd-free: Podman supervises it via
|
||||
`restart: unless-stopped`.
|
||||
- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off; a profile
|
||||
already exists in `/data` from the gateway.
|
||||
- The healthcheck `start_period` is 600 s because the very first boot spends
|
||||
minutes pip-installing into a fresh venv.
|
||||
|
||||
## Remote access (Hermex on iOS, from anywhere)
|
||||
|
||||
Only port `22` is forwarded to Dankclaw at the router, so the webui is
|
||||
reached through an SSH local forward — no extra router ports, no third-party
|
||||
relay (no Tailscale, no Cloudflare):
|
||||
|
||||
1. On the iPhone, open **Termius** (or any SSH client with local port
|
||||
forwarding) and connect to `<dankclaw-public>:22` with the usual key.
|
||||
2. On that connection, add a **local forward**: `127.0.0.1:8787` →
|
||||
`127.0.0.1:8787`. The forward runs inside the encrypted SSH session;
|
||||
nothing else on the phone routes through it.
|
||||
3. In Hermex, set the server URL to `http://localhost:8787` and sign in with
|
||||
the webui password.
|
||||
|
||||
The tunnel lives as long as the SSH session on the phone — iOS does not allow
|
||||
a true background daemon, so connect before you use Hermex.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Hermex app] -->|http://localhost:8787| B[SSH local forward on iPhone]
|
||||
B ==>|encrypted SSH to public IP:22| C[Dankclaw sshd]
|
||||
C --> D["hermes-webui 127.0.0.1:8787"]
|
||||
D --> E["/home/dank/hermes-data (shared agent state)"]
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
./scripts/deploy.sh dankclaw --service hermes-webui
|
||||
```
|
||||
|
||||
The planner maps changes to the `hermes-webui` compose block (and only that
|
||||
block) to this service, so gateway and inference restarts stay isolated.
|
||||
Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
|
||||
`dankclaw/.env`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The `/webui` mount is **rw on purpose**: `bootstrap.py` creates `.venv` in
|
||||
the repo root. Do not flip it to `:ro`.
|
||||
- Never publish `8787` beyond loopback without keeping the password set —
|
||||
the webui can run agent commands.
|
||||
- The webui and the gateway share one data volume. A config change made on one
|
||||
side takes effect when the other restarts (same caveat as the dashboard).
|
||||
- `bootstrap.py` auto-installs a Hermes Agent if it cannot find one; in this
|
||||
image the agent is already present, so that path should never fire. If the
|
||||
log shows an install attempt, the image changed — re-verify the entrypoint.
|
||||
|
||||
# Citations
|
||||
|
||||
- hermes-webui README (env vars, `--foreground`, `/health`): https://github.com/nesquena/hermes-webui
|
||||
- Hermex setup model (server URL + password; Cloudflare/Tailscale are optional, not required): https://hermexapp.com/
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes]] — the agent and its Signal gateway
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,52 @@
|
||||
# Hermes
|
||||
|
||||
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `hermes.dankclaw.local` — **Docs:** https://hermes-agent.nousresearch.com/docs
|
||||
|
||||
**Persistent state:** `/home/dank/hermes-data` on Dankclaw, mounted as `/data` in
|
||||
both containers. The tracked, sanitized configuration baseline is
|
||||
[`dankclaw/hermes/config.yaml.example`](../../dankclaw/hermes/config.yaml.example).
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
|
||||
| License | MIT |
|
||||
| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
|
||||
| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
|
||||
| Memory | FTS5 session search, Honcho user modeling |
|
||||
| Skills | self-created, self-improving during use |
|
||||
|
||||
## Configuration
|
||||
|
||||
- Point Hermes at the local model endpoint or an external provider through its provider config.
|
||||
- The gateway process exposes one port for chat integration; the exact port is set in the Hermes config (`8090` convention in this repo).
|
||||
- The data volume persists skills, memory, and session history.
|
||||
- Hermes has two named local providers: `dankclaw-large` reaches
|
||||
`http://192.168.0.143:7770/v1` and exposes `frank-default-large`;
|
||||
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
|
||||
`frank-default-small`.
|
||||
- The live `config.yaml` contains secrets and user state and is not committed;
|
||||
the tracked example is the sanitized backup of its non-secret settings.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/data` | `/home/dank/hermes-data` |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Hermes is under active development; pin the image tag for reproducibility.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankclaw, 7770) is the intended backend.
|
||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
- [[Hermes-WebUI]] — the Hermex server (phone-facing API, SSH-tunneled)
|
||||
- [[Dankclaw Stack]]
|
||||
- [[OpenWebUI]] — the chat UI that pairs with Hermes
|
||||
- Knowledge base: [[SearXNG]] (Hermes as the search agent)
|
||||
@@ -0,0 +1,75 @@
|
||||
# OpenWebUI
|
||||
|
||||
OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compatible endpoints (vLLM, llama.cpp, Ollama) and adds RAG, plugins, tool calling, and multi-user support.
|
||||
|
||||
**Host:** Dankclaw — **Port:** 3005 (host) → 8080 (container) — **Boot-persistent:** Quadlet — **Docs:** https://github.com/open-webui/open-webui
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Default backend | CIRU llama.cpp on Dankclaw `:7770` = `frank-default-large` (via host LAN IP `192.168.0.143:7770`) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
|
||||
## The LLM backends
|
||||
|
||||
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
|
||||
|
||||
| Endpoint | Model name | Host | Capability |
|
||||
|----------|-----------|------|-----------|
|
||||
| `:7770` (CIRU llama.cpp) | `frank-default-large` | Dankclaw | completion |
|
||||
| `:7770` (llama.cpp) | `frank-default-small` | Dankserver | completion, multimodal |
|
||||
|
||||
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
|
||||
|
||||
> Both endpoints are bare-metal systemd services managed by this repository (the former `vllmScripts` repo, merged in 2026-08-22). Model swaps, context sizing, and spec tuning happen here. See the [LLM knowledge base](../LLM/index.md).
|
||||
|
||||
## Boot persistence (Quadlet)
|
||||
|
||||
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|----------------|-----------|
|
||||
| `/app/backend/data` | `/home/dank/Containers/open-webui/data` |
|
||||
|
||||
The data dir holds the database, RAG indexes, and uploads. The deploy script creates it if missing and preserves it across reruns.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start period). It checks two things from inside the container:
|
||||
|
||||
1. **App liveness** — `GET http://127.0.0.1:8080/health` returns 200 (`{"status":true}`).
|
||||
2. **Network liveness** — a non-loopback interface exists in `/sys/class/net`. The dead-tap zombie state (History) leaves only `lo` behind, and this is the check that catches it.
|
||||
|
||||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||||
|
||||
## Retired instances
|
||||
|
||||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||||
- The bare-container `open-webui` on Dankserver (port 3000) pointed at a dead backend (`:9090`) and was removed. Its data dir is preserved on Dankserver in case it is needed.
|
||||
- The bare-container `searxng` on Dankserver (port 8081) was removed. SearXNG now runs only on Dankclaw behind Caddy.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Use the non-CUDA image on this CPU-only host.
|
||||
- The default admin account is created on first login; secure it immediately.
|
||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||||
- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||
- [[SearXNG]] — the search engine on this host
|
||||
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
|
||||
@@ -0,0 +1,70 @@
|
||||
# Postgres
|
||||
|
||||
PostgreSQL is the database backend for Gitea. It runs on the same podman network as Gitea and holds the repository metadata, users, and issue data.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://www.postgresql.org/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/library/postgres:16-alpine` |
|
||||
| Port | 5432 (internal only) |
|
||||
| Data volume | bind mount at `/var/lib/postgresql/data` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: docker.io/library/postgres:16-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: gitea
|
||||
POSTGRES_PASSWORD: "${GITEA_DB_PASSWORD}"
|
||||
POSTGRES_DB: gitea
|
||||
volumes:
|
||||
- /opt/gitea/data/gitea-postgres:/var/lib/postgresql/data
|
||||
```
|
||||
|
||||
## Gitea Connection
|
||||
|
||||
Gitea reaches the database over the `systemd-gitea` network (quadlet-managed live, compose-managed after the full-stack deploy):
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
DATABASE_TYPE: postgres
|
||||
DATABASE_HOST: gitea-postgres
|
||||
DATABASE_NAME: gitea
|
||||
DATABASE_USER: gitea
|
||||
DATABASE_PASSWD: "${GITEA_DB_PASSWORD}"
|
||||
```
|
||||
|
||||
The live `app.ini` on the host overrides env vars. When you rename the database container, update its `[database] HOST` line. See [[Gitea]] Gotchas.
|
||||
|
||||
## Migration
|
||||
|
||||
The live Gitea stack on Dankclaw already runs this image with the data at `/opt/gitea/data/gitea-postgres`. The migration keeps the bind mount in place; see [[Gitea]].
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Use `pg_isready`:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gitea"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The database password must match between the `postgres` service and Gitea's `DATABASE_PASSWD`; both come from the same `.env` variable.
|
||||
- Do not change the Gitea `SECRET_KEY` or `INTERNAL_TOKEN` after migration — encrypted data becomes unreadable.
|
||||
- Back up with `pg_dump`; see [[Gitea]] (knowledge base).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Gitea]]
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,400 @@
|
||||
# SearXNG Providers
|
||||
|
||||
How Google and other upstream search engines get blocked behind the Gluetun
|
||||
VPN tunnel, and how to unblock them in SearXNG.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://docs.searxng.org
|
||||
|
||||
SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
|
||||
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
|
||||
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
|
||||
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
|
||||
engine for an hour to 15 days and silently drops it from results. That is the
|
||||
"Google (and sometimes a few others) went unavailable" symptom.
|
||||
|
||||
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
|
||||
are the detail if you need it.
|
||||
|
||||
## Provider policy
|
||||
|
||||
The default aggregation should maximize **independent, anonymous coverage**
|
||||
without turning one user query into a request storm. Disabled engines remain
|
||||
installed and selectable on demand; `inactive` is reserved for engines that
|
||||
should disappear completely.
|
||||
|
||||
| Category | Default baseline | On-demand / deferred |
|
||||
|----------|------------------|----------------------|
|
||||
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
|
||||
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
|
||||
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
|
||||
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
|
||||
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
|
||||
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
|
||||
|
||||
This policy uses no API credentials and does not identify the operator to a
|
||||
search provider. It is deliberately conservative: a bundled engine is not
|
||||
necessarily healthy, anonymous, or suitable for every default query.
|
||||
|
||||
## Root cause (one line)
|
||||
|
||||
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
|
||||
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
|
||||
rate-limit it → CAPTCHA → engine **suspension** → empty results.
|
||||
|
||||
## The suspension model (what "unavailable" actually is)
|
||||
|
||||
An engine that returns 429 / reCAPTCHA / access-denied raises a named
|
||||
exception and is *suspended* (removed from result aggregation for a set time).
|
||||
Suspended engines are the usual cause of "gone":
|
||||
|
||||
| Exception | Default suspension | Trigger |
|
||||
|-----------|--------------------|---------|
|
||||
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
|
||||
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
|
||||
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
|
||||
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
|
||||
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
|
||||
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
|
||||
|
||||
The values are `search.suspended_times` (+ `ban_time_on_fail`) in
|
||||
`settings.yml`. An engine that returns **0 results without an error** is *not*
|
||||
suspended — that is the separate "selector drift" problem (below).
|
||||
|
||||
## Two failure modes
|
||||
|
||||
| Mode | Symptom | Root cause | Fix family |
|
||||
|------|---------|-----------|-----------|
|
||||
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
|
||||
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |
|
||||
|
||||
Most "Google is broken on a VPN host" cases are a **mix of both**.
|
||||
|
||||
## Fix A — Engine config (settings.yml `engines:`)
|
||||
|
||||
The current tracked policy enables Bing and DuckDuckGo for default general
|
||||
search, gives each weight 2, disables pagination, and allows 5 seconds for the
|
||||
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
|
||||
but disabled from default aggregation. This limits fan-out while preserving
|
||||
explicit/on-demand selection.
|
||||
|
||||
Important: SearXNG's engines are configured in the **`engines:` list inside
|
||||
`settings.yml`**, not a separate `engines.yml` file. With
|
||||
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
|
||||
the built-in engines by `name` — you do **not** set `use_default_settings: false`.
|
||||
|
||||
To add, override, disable, or prune:
|
||||
|
||||
```yaml
|
||||
# shared/searxng/settings.yml (appended to the existing file)
|
||||
use_default_settings: true
|
||||
engines:
|
||||
# prune a broken engine entirely
|
||||
remove:
|
||||
- # (example) wikipedia # drop an engine that never recovers
|
||||
|
||||
# override + duplicate the Google engines so one survives the other's suspension
|
||||
- name: google
|
||||
weight: 3
|
||||
disabled: false
|
||||
timeout: 5.0 # wait longer over the tunnel (global default is 2.0s)
|
||||
retry_on_http_error: [403, 429]
|
||||
display_error_messages: true
|
||||
|
||||
- name: google2 # fresh engine, same module, survives google's CAPTCHA ban
|
||||
engine: google
|
||||
categories: general
|
||||
weight: 2
|
||||
timeout: 5.0
|
||||
disabled: false
|
||||
```
|
||||
|
||||
Key per-engine keys (each overwrites the engine-module default; see
|
||||
`settings_engines.html`):
|
||||
|
||||
| Key | Effect |
|
||||
|-----|--------|
|
||||
| `name` | merge key (lowercase, no underscore) |
|
||||
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
|
||||
| `categories` | categories it serves (also a comma string) |
|
||||
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
|
||||
| `weight` | ranks results higher |
|
||||
| `disabled` | off by default, selectable in /preferences |
|
||||
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
|
||||
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
|
||||
| `display_error_messages` | show/hide the engine's error banner |
|
||||
| `proxies` | per-engine egress override (see Fix C) |
|
||||
| `language` | force region, e.g. `en` |
|
||||
|
||||
**Best duplicate trick:** re-declare `google` under a new `name` in a
|
||||
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
|
||||
`google2` still returns results, and vice-versa.
|
||||
|
||||
**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
|
||||
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
|
||||
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
|
||||
[scrapy_html]` under `name: google` mostly does nothing. To make Google
|
||||
robust against selector drift, either update the upstream `google.py`
|
||||
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
|
||||
Verify `parsers` consumption on the pinned version before depending on it.
|
||||
|
||||
**JSON-feed fallback:** decouple parsing from Google's HTML churn.
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
- name: google-json
|
||||
engine: json_engine
|
||||
categories: general
|
||||
paging: true
|
||||
search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
|
||||
results_query: results
|
||||
title_query: title
|
||||
url_query: link
|
||||
content_query: snippet
|
||||
```
|
||||
|
||||
## Fix B — Request tuning (settings.yml `outgoing:`)
|
||||
|
||||
SearXNG already rotates a **random browser User-Agent per outgoing request**
|
||||
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
|
||||
key**. The blogs that point at a `useragents:` table describe the old
|
||||
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.
|
||||
|
||||
Tune the `outgoing` block and the suspend model to survive load:
|
||||
|
||||
```yaml
|
||||
outgoing:
|
||||
request_timeout: 4.0 # current Dankclaw value; modest VPN-latency allowance
|
||||
max_request_timeout: 12.0
|
||||
pool_connections: 100
|
||||
pool_maxsize: 20
|
||||
enable_http2: true
|
||||
retries: 1 # avoid amplifying 429/CAPTCHA responses from one exit IP
|
||||
useragent_suffix: "ops@yourhomelab.example" # so a per-engine blocker can reach you
|
||||
|
||||
search:
|
||||
ban_time_on_fail: 10
|
||||
max_ban_time_on_fail: 180
|
||||
suspended_times:
|
||||
SearxEngineTooManyRequests: 300 # 429 → 5 min before it retries (tune down to recover faster)
|
||||
SearxEngineCaptcha: 7200 # CAPTCHA → 2h before it retries
|
||||
```
|
||||
|
||||
Notes:
|
||||
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
|
||||
`time_between_requests` sleep field does **not** exist in current SearXNG —
|
||||
that is a historical-searx concept. The functional equivalent is the suspend
|
||||
schedule above (widen to back off before blocking, narrow to recover faster).
|
||||
- `request_timeout` interacts with the suspend model: too tight → engines look
|
||||
"unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
|
||||
12 seconds as the cap; raise it only when measurements show real upstream
|
||||
latency rather than 429/CAPTCHA responses.
|
||||
- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
|
||||
provider's rate limit worse. A retry count of 1 is a bounded compromise for a
|
||||
transient network failure.
|
||||
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
|
||||
engine return zero rows and *look* blocked.
|
||||
|
||||
## Fix C — Routing & proxies (change which IP the upstreams exit as)
|
||||
|
||||
Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
|
||||
strategy. The decisive fact for our topology: a proxy that runs **inside the
|
||||
same Gluetun netns** exits from the *same* single IP, so round-robin among
|
||||
same-IP proxies **does not** unblock. The proxy must egress from a **different,
|
||||
clean IP**.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
|
||||
SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
|
||||
```
|
||||
|
||||
**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
|
||||
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
|
||||
clean IP:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
- name: google
|
||||
proxies:
|
||||
http:
|
||||
- http://gluetun-upstream:8080
|
||||
https:
|
||||
- http://gluetun-upstream:8080
|
||||
- socks5h://gluetun-upstream:8388
|
||||
- name: google news
|
||||
engine: google_news
|
||||
proxies:
|
||||
https:
|
||||
- http://gluetun-upstream:8080
|
||||
```
|
||||
|
||||
`using_tor_proxy` follows the same per-engine/global pattern.
|
||||
|
||||
**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
|
||||
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
|
||||
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
|
||||
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
|
||||
Gives a second isolated exit IP and reputation pool.
|
||||
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
|
||||
moves to Gluetun, and **split-brain DNS** (below).
|
||||
|
||||
**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
|
||||
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
|
||||
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
|
||||
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
|
||||
via `SERVER_HOSTNAMES`.
|
||||
|
||||
**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
|
||||
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
|
||||
rotating the **shared** Gluetun rotates the exit for *all* attached
|
||||
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
|
||||
gateway.
|
||||
|
||||
> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
|
||||
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
|
||||
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
|
||||
> containers/systemd/<unit>.container` `Environment` to inject names into the
|
||||
> tunnel's `/etc/hosts`.
|
||||
|
||||
## Observed Dankclaw baseline (2026-08-25)
|
||||
|
||||
The read-only audit found:
|
||||
|
||||
- SearXNG image: `ghcr.io/searxng/searxng:latest`; embedded version `1.0.0`.
|
||||
- Gluetun uses `VPN_SERVICE_PROVIDER=custom` with a static WireGuard profile;
|
||||
no `SERVER_COUNTRIES`, `SERVER_CITIES`, `SERVER_REGIONS`, or
|
||||
`SERVER_HOSTNAMES` filters are configured.
|
||||
- The observed exit was `149.22.88.19` (Seattle, Washington, US).
|
||||
- Bing and DuckDuckGo returned results. Mojeek returned no results without an
|
||||
upstream error.
|
||||
- `google cse` and Brave were suspended for too many requests; Startpage was
|
||||
suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded
|
||||
Google's "unusual traffic from your network" response.
|
||||
- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
|
||||
healthy. This isolates the primary failures to upstream provider handling,
|
||||
not application availability.
|
||||
|
||||
This deployment returns the legacy array form for `unresponsive_engines`:
|
||||
|
||||
```json
|
||||
"unresponsive_engines": [
|
||||
["brave", "Suspended: too many requests"],
|
||||
["startpage", "Suspended: CAPTCHA"]
|
||||
]
|
||||
```
|
||||
|
||||
Do not assume the newer object form (`engine`, `error_type`, `suspended`)
|
||||
until the live response has been checked. The direct published port is the
|
||||
useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
|
||||
hostname resolved to the WAN address during the audit and timed out from the
|
||||
audit host, so it was not used as the baseline path.
|
||||
|
||||
## Fix D — Verify an engine is unblocked
|
||||
|
||||
Three layers, cheapest first. All queries should hit the path production uses
|
||||
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
|
||||
|
||||
**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
|
||||
(otherwise `format=json` → 403), then query a single engine:
|
||||
|
||||
```bash
|
||||
BASE="http://192.168.0.143:8081" # published port (ProtonVPN exit)
|
||||
curl -s "$BASE/search?q=test&engines=google&format=json" \
|
||||
| jq -r '.unresponsive_engines[]? | select(.engine=="google")'
|
||||
```
|
||||
|
||||
Inspect:
|
||||
- `content[].engine` — which engine produced each result.
|
||||
- `unresponsive_engines[]` — inspect the live shape first. On Dankclaw it is
|
||||
`[engine, message]`, for example `["google cse", "Suspended: too many requests"]`.
|
||||
Newer builds may return objects containing `engine`, `error_type`, and
|
||||
`suspended`.
|
||||
- `timings[].engine` — per-engine latency when the running build emits timings.
|
||||
|
||||
Full status check (bash + jq):
|
||||
|
||||
```bash
|
||||
BASE="http://192.168.0.143:8081"
|
||||
for e in google bing duckduckgo brave startpage; do
|
||||
curl -s "$BASE/search?q=test&engines=$e&format=json" \
|
||||
| jq -r --arg e "$e" '
|
||||
(.unresponsive_engines // [])
|
||||
| map(select(.engine==$e))
|
||||
| if length==0 then "\($e): OK"
|
||||
else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
|
||||
done
|
||||
```
|
||||
|
||||
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
|
||||
categories — an inventory of what is live. `?engines=<name>` isolates a single
|
||||
provider; `/debug` adds engine diagnostics.
|
||||
|
||||
**3. `check.searx.space`** — the hosted official checker probes every engine of
|
||||
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
|
||||
plain HTTP probe misses) and posts results on searx.space. Add your instance via
|
||||
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
|
||||
to reach the SearXNG port.
|
||||
|
||||
Local self-test CLI (`kvch/searx-checker`):
|
||||
|
||||
```bash
|
||||
python3 checker/checker.py http://192.168.0.143:8081
|
||||
# google.OK bing.OK gitlab......ERROR
|
||||
```
|
||||
|
||||
Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
|
||||
signal), not a direct host port (which uses the host LAN IP and can be
|
||||
misleading).
|
||||
|
||||
## Recommended path for Dankclaw (ranked)
|
||||
|
||||
1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
|
||||
query classes before enabling a bundled engine in default aggregation.
|
||||
2. **Keep default fan-out small** — prefer independent anonymous providers and
|
||||
disable pagination for rate-sensitive engines.
|
||||
3. **Promote by evidence** — raise `weight` only for engines that return useful
|
||||
results consistently; weight is not a health check.
|
||||
4. **Avoid credentials** — do not add API-backed engines unless the provider is
|
||||
genuinely free, anonymous, and compatible with our privacy requirements.
|
||||
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
|
||||
exactly which engines are down and why. Confirm the running SearXNG version
|
||||
(check whether `parsers`/`rate_limit` exist in the pinned build).
|
||||
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
|
||||
upstream request, caps it at 15 seconds, and retries once. This addresses
|
||||
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
|
||||
after measuring the resulting error and latency profile.
|
||||
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
|
||||
a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
|
||||
isolation is needed). Avoid the same-netns-proxy trap.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
|
||||
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
|
||||
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
|
||||
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
|
||||
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
|
||||
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
|
||||
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
|
||||
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
|
||||
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
|
||||
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
|
||||
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
|
||||
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
|
||||
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
|
||||
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
|
||||
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
|
||||
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
|
||||
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
|
||||
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
|
||||
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
|
||||
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the stack, routing, limiter, health
|
||||
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
|
||||
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
|
||||
- [[Service Research Brief]] — the research that produced these pages
|
||||
@@ -0,0 +1,151 @@
|
||||
# SearXNG
|
||||
|
||||
SearXNG is a privacy-respecting metasearch engine. It aggregates results from many upstream search engines into one result set and serves both an HTML UI and a JSON API for agents.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `search.franksdank.cloud` (Caddy) — **Docs:** https://docs.searxng.org
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
|
||||
| Port | 8081 (host, on Gluetun) → 8080 (container) |
|
||||
| Config | `/etc/searxng/settings.yml` (bind mount) |
|
||||
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
|
||||
| Formats | `html`, `json` (agent API) |
|
||||
|
||||
## VPN Routing (critical)
|
||||
|
||||
All SearXNG outbound traffic exits through the Gluetun VPN container. The searxng unit sets `Network=gluetun.container`, which makes it reuse gluetun's network namespace:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
```
|
||||
|
||||
Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]] (KB Pattern 2 is the one used here).
|
||||
|
||||
## Boot Persistence (Quadlet)
|
||||
|
||||
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
|
||||
|
||||
| Unit file | Service | Role |
|
||||
|-----------|---------|------|
|
||||
| `searxng.network` | `searxng-network.service` | DNS-enabled bridge, subnet pinned to 10.89.2.0/24 |
|
||||
| `gluetun.container` | `gluetun.service` | WireGuard tunnel; publishes 8081:8080 |
|
||||
| `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache, alias `valkey` |
|
||||
| `searxng.container` | `searxng.service` | Reuses gluetun's netns; publishes no port |
|
||||
|
||||
gluetun and valkey join the searxng network. searxng reuses gluetun's netns and reaches valkey by name on that network. Each unit sets `WantedBy=default.target`. User linger is on, so the units start at boot.
|
||||
|
||||
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
|
||||
|
||||
## settings.yml Keys
|
||||
|
||||
```yaml
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "${SEARXNG_SECRET}" # from .env
|
||||
base_url: "${SEARXNG_BASE_URL}" # resolves to http://192.168.0.143:8081/
|
||||
limiter: true
|
||||
method: "POST"
|
||||
|
||||
search:
|
||||
default_lang: "en"
|
||||
formats:
|
||||
- html
|
||||
- json
|
||||
|
||||
valkey:
|
||||
url: "valkey://searxng-valkey:6379/0"
|
||||
|
||||
outgoing:
|
||||
request_timeout: 5.0
|
||||
max_request_timeout: 15.0
|
||||
retries: 1 # avoid amplifying upstream 429/CAPTCHA responses
|
||||
|
||||
# Anonymous, bounded default aggregation. Disabled engines remain configured
|
||||
# for explicit/on-demand selection; see SearXNG Providers for the policy.
|
||||
engines:
|
||||
- name: bing
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: duckduckgo
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: google cse
|
||||
disabled: true
|
||||
- name: brave
|
||||
disabled: true
|
||||
- name: startpage
|
||||
disabled: true
|
||||
```
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/searxng` | `/opt/searxng/config` |
|
||||
| `/etc/searxng/valkey` | (valkey container data) |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
SearXNG responds on `/healthz` with 200 when ready.
|
||||
|
||||
## Trusted Group (limiter.toml)
|
||||
|
||||
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
|
||||
|
||||
Two lists control access:
|
||||
|
||||
- `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
|
||||
- `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).
|
||||
|
||||
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
|
||||
|
||||
Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.
|
||||
|
||||
The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.
|
||||
|
||||
Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
|
||||
|
||||
## Unblocking Google & Search Providers
|
||||
|
||||
The current request policy allows a modest amount of VPN latency (`4.0s` per
|
||||
request, `12.0s` maximum) while limiting retries to one. This can recover slow
|
||||
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
|
||||
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
|
||||
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
|
||||
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
|
||||
disables observed CAPTCHA/429 providers from normal aggregation, and disables
|
||||
pagination to reduce upstream request volume. SearXNG **suspends** offending
|
||||
engines for hours to days, which is what shows up as "Google went unavailable".
|
||||
This is upstream, not local.
|
||||
|
||||
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
|
||||
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
|
||||
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
|
||||
if Google still re-blocks — route the Google engines at a clean-IP proxy via
|
||||
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
|
||||
root cause, the suspend-model table, the `engines:` config, and the
|
||||
`/json` verification recipe.
|
||||
|
||||
### Gotchas
|
||||
|
||||
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
|
||||
- The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
|
||||
- The config file is root-owned (uid of the rootless podman namespace). Update it with a one-off `sudo cp`, then run `systemctl --user restart searxng.service` to reload.
|
||||
- `base_url` must match the public address or proxy rewrites break.
|
||||
- Keep the valkey cache container on the same podman network as searxng.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Gluetun]] — the VPN container
|
||||
- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
|
||||
- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]
|
||||
@@ -0,0 +1,55 @@
|
||||
# Valkey
|
||||
|
||||
Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://valkey.io
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `valkey/valkey:9-alpine` |
|
||||
| Port | 6379 (internal, on the SearXNG network) |
|
||||
| Role | SearXNG limiter + result cache |
|
||||
|
||||
## Why Valkey
|
||||
|
||||
The official SearXNG docker-compose ships a `searxng-valkey` container. Valkey is the Redis fork under the Linux Foundation umbrella; the SearXNG project moved to it as the default cache.
|
||||
|
||||
## Configuration
|
||||
|
||||
The valkey unit is `searxng-valkey.container` (see [[SearXNG]] for the trio). It joins the searxng network and is reachable by the name `searxng-valkey` (container name) and the alias `valkey`:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng-valkey
|
||||
Network=searxng.network
|
||||
NetworkAlias=valkey
|
||||
Image=docker.io/valkey/valkey:9-alpine
|
||||
Volume=/opt/searxng/valkey:/data
|
||||
```
|
||||
|
||||
SearXNG connects to it via `settings.yml`:
|
||||
|
||||
```yaml
|
||||
valkey:
|
||||
url: "valkey://searxng-valkey:6379/0"
|
||||
```
|
||||
|
||||
It sets `WantedBy=default.target`. User linger is on, so it starts at boot.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/data` | `/opt/searxng/valkey` |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart.
|
||||
- Keep Valkey on the same podman network as SearXNG.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the consumer
|
||||
- [[Dankclaw Stack]]
|
||||
Reference in new issue
Block a user