152 lines
7.2 KiB
Markdown
152 lines
7.2 KiB
Markdown
# SearXNG
|
|
|
|
SearXNG is a privacy-respecting metasearch engine. It aggregates results from many upstream search engines into one result set and serves both an HTML UI and a JSON API for agents.
|
|
|
|
**Host:** Dankclaw — **Hostname:** `search.franksdank.cloud` (Caddy) — **Docs:** https://docs.searxng.org
|
|
|
|
## Facts (from research, primary source)
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
|
|
| Port | 8081 (host, on Gluetun) → 8080 (container) |
|
|
| Config | `/etc/searxng/settings.yml` (bind mount) |
|
|
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
|
|
| Formats | `html`, `json` (agent API) |
|
|
|
|
## VPN Routing (critical)
|
|
|
|
All SearXNG outbound traffic exits through the Gluetun VPN container. The searxng unit sets `Network=gluetun.container`, which makes it reuse gluetun's network namespace:
|
|
|
|
```ini
|
|
[Container]
|
|
ContainerName=searxng
|
|
Network=gluetun.container
|
|
```
|
|
|
|
Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]] (KB Pattern 2 is the one used here).
|
|
|
|
## Boot Persistence (Quadlet)
|
|
|
|
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
|
|
|
|
| Unit file | Service | Role |
|
|
|-----------|---------|------|
|
|
| `searxng.network` | `searxng-network.service` | DNS-enabled bridge, subnet pinned to 10.89.2.0/24 |
|
|
| `gluetun.container` | `gluetun.service` | WireGuard tunnel; publishes 8081:8080 |
|
|
| `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache, alias `valkey` |
|
|
| `searxng.container` | `searxng.service` | Reuses gluetun's netns; publishes no port |
|
|
|
|
gluetun and valkey join the searxng network. searxng reuses gluetun's netns and reaches valkey by name on that network. Each unit sets `WantedBy=default.target`. User linger is on, so the units start at boot.
|
|
|
|
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
|
|
|
|
## settings.yml Keys
|
|
|
|
```yaml
|
|
use_default_settings: true
|
|
server:
|
|
secret_key: "${SEARXNG_SECRET}" # from .env
|
|
base_url: "${SEARXNG_BASE_URL}" # resolves to http://192.168.0.143:8081/
|
|
limiter: true
|
|
method: "POST"
|
|
|
|
search:
|
|
default_lang: "en"
|
|
formats:
|
|
- html
|
|
- json
|
|
|
|
valkey:
|
|
url: "valkey://searxng-valkey:6379/0"
|
|
|
|
outgoing:
|
|
request_timeout: 5.0
|
|
max_request_timeout: 15.0
|
|
retries: 1 # avoid amplifying upstream 429/CAPTCHA responses
|
|
|
|
# Anonymous, bounded default aggregation. Disabled engines remain configured
|
|
# for explicit/on-demand selection; see SearXNG Providers for the policy.
|
|
engines:
|
|
- name: bing
|
|
disabled: false
|
|
weight: 2
|
|
paging: false
|
|
timeout: 5.0
|
|
- name: duckduckgo
|
|
disabled: false
|
|
weight: 2
|
|
paging: false
|
|
timeout: 5.0
|
|
- name: google cse
|
|
disabled: true
|
|
- name: brave
|
|
disabled: true
|
|
- name: startpage
|
|
disabled: true
|
|
```
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path (convention) |
|
|
|----------------|------------------------|
|
|
| `/etc/searxng` | `/opt/searxng/config` |
|
|
| `/etc/searxng/valkey` | (valkey container data) |
|
|
|
|
## Healthcheck
|
|
|
|
SearXNG responds on `/healthz` with 200 when ready.
|
|
|
|
## Trusted Group (limiter.toml)
|
|
|
|
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
|
|
|
|
Two lists control access:
|
|
|
|
- `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
|
|
- `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).
|
|
|
|
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
|
|
|
|
Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.
|
|
|
|
The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.
|
|
|
|
Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
|
|
|
|
## Unblocking Google & Search Providers
|
|
|
|
The current request policy allows a modest amount of VPN latency (`4.0s` per
|
|
request, `12.0s` maximum) while limiting retries to one. This can recover slow
|
|
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
|
|
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
|
|
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
|
|
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
|
|
disables observed CAPTCHA/429 providers from normal aggregation, and disables
|
|
pagination to reduce upstream request volume. SearXNG **suspends** offending
|
|
engines for hours to days, which is what shows up as "Google went unavailable".
|
|
This is upstream, not local.
|
|
|
|
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
|
|
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
|
|
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
|
|
if Google still re-blocks — route the Google engines at a clean-IP proxy via
|
|
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
|
|
root cause, the suspend-model table, the `engines:` config, and the
|
|
`/json` verification recipe.
|
|
|
|
### Gotchas
|
|
|
|
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
|
|
- The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
|
|
- The config file is root-owned (uid of the rootless podman namespace). Update it with a one-off `sudo cp`, then run `systemctl --user restart searxng.service` to reload.
|
|
- `base_url` must match the public address or proxy rewrites break.
|
|
- Keep the valkey cache container on the same podman network as searxng.
|
|
|
|
## See Also
|
|
|
|
- [[Dankclaw Stack]]
|
|
- [[Gluetun]] — the VPN container
|
|
- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
|
|
- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]
|