serverAppConfigs/serverAppConfigs-docs/Dankclaw/SearXNG.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	SearXNG is a privacy-respecting metasearch engine.	dankclaw searxng	2026-09-03

SearXNG

SearXNG is a privacy-respecting metasearch engine. It aggregates results from many upstream search engines into one result set and serves both an HTML UI and a JSON API for agents.

Host: Dankclaw — Hostname: search.franksdank.cloud (Caddy) — Docs: https://docs.searxng.org

Facts (from research, primary source)

Item	Value
Image	ghcr.io/searxng/searxng:latest (or DockerHub mirror)
Port	8081 (host, on Gluetun) → 8080 (container)
Config	/etc/searxng/settings.yml (bind mount)
Cache	Valkey (replaces Redis) — valkey/valkey:9-alpine
Formats	html, json (agent API)

VPN Routing (critical)

All SearXNG outbound traffic exits through the Gluetun VPN container. The searxng unit sets Network=gluetun.container, which makes it reuse gluetun's network namespace:

[Container]
ContainerName=searxng
Network=gluetun.container

Port publishing happens on the gluetun unit, not on searxng. See Gluetun (KB Pattern 2 is the one used here).

Boot Persistence (Quadlet)

The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in searxng/quadlet/ and install to ~/.config/containers/systemd/.

The units live in dankclaw/searxng/quadlet/ and install to ~/.config/containers/systemd/ on Dankclaw (verified live):

Unit file	Service	Role
searxng.network	generated network service	DNS-enabled bridge; the created network is named systemd-searxng, subnet pinned to 10.89.2.0/24
gluetun.container	gluetun.service	WireGuard tunnel (env-file secrets); publishes 8081:8080 — the only published port in the trio
searxng-valkey.container	searxng-valkey.service	Valkey cache on the searxng network, alias valkey (see Valkey)
searxng.container	searxng.service	Network=gluetun.container — joins the tunnel's netns, so it publishes no port of its own; reached through gluetun's 8081

gluetun and valkey join the searxng network. searxng reuses gluetun's netns and reaches valkey by name on that network. Each unit sets WantedBy=default.target. User linger is on, so the units start at boot.

Deploy with scripts/deploy-searxng-quadlet.sh dankclaw. It copies the units, daemon-reloads, enables them, and starts them in dependency order.

settings.yml Keys

use_default_settings: true
server:
  secret_key: "${SEARXNG_SECRET}"   # rendered at deploy by scripts/searxng-render-settings.py
  base_url: "${SEARXNG_BASE_URL}"   # same renderer; SearXNG does NOT expand env vars itself
  limiter: true
  method: "POST"

search:
  default_lang: "en"
  formats:
    - html
    - json

valkey:
  url: "valkey://searxng-valkey:6379/0"

outgoing:
  request_timeout: 5.0
  max_request_timeout: 15.0
  retries: 1                     # avoid amplifying upstream 429/CAPTCHA responses

# Anonymous, bounded default aggregation. Disabled engines remain configured
# for explicit/on-demand selection; see SearXNG Providers for the policy.
engines:
  - name: bing
    disabled: false
    weight: 2
    paging: false
    timeout: 5.0
  - name: duckduckgo
    disabled: false
    weight: 2
  - name: kagi
    engine: kagi
    categories: [general, web]
    shortcut: kg
    api_key: "${KAGI_API_KEY}"      # rendered at deploy from /opt/searxng/.env
    kagi_categ: search
    disabled: false
    weight: 2
    paging: false                  # one API credit per aggregated query
    timeout: 8.0
  - name: google cse
    disabled: true
  - name: brave
    disabled: true
  - name: startpage
    disabled: true

Volumes

Container path	Host path (convention)
/etc/searxng	/opt/searxng/config
/etc/searxng/valkey	(valkey container data)

Healthcheck

SearXNG responds on /healthz with 200 when ready.

Trusted Group (limiter.toml)

The trusted group lives in shared/searxng/limiter.toml, not in settings.yml. The settings.yml file only turns the limiter on. The file deploys to /opt/searxng/config/limiter.toml, mounted at /etc/searxng/limiter.toml.

Two lists control access:

• pass_ip — an IP in this list gets unrestricted access and bypasses all bot detection.
• trusted_proxies — if the request IP is in this list, SearXNG reads the real client IP from X-Forwarded-For (set by Caddy).

The pass_ip list holds loopback (127.0.0.1/32, ::1), the whole homelab LAN (192.168.0.0/24), and the host's private podman/slirp ranges (10.88.0.0/16, 10.89.0.0/16). An IP in pass_ip skips every bot-detection check, including the rate limiter. deploy.sh copies the file into place on each deploy.

Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (10.88.x/10.89.x), never the real client IP. Only Caddy's X-Forwarded-For restores it. So a local subagent that hits the port directly must present a pass_ip source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.

The rate limiter's JSON cap is API_MAX = 4 requests per IP per hour. The constant is hardcoded in Python (searx/botdetection/ip_limit.py). It cannot be raised from limiter.toml. pass_ip is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.

Reach SearXNG through Caddy (search.franksdank.cloud, guarded) for browser use. A local subagent can hit the published port directly because its slirp/loopback/LAN sources are in pass_ip. The searxng network (systemd-searxng, 10.89.2.0/24) is in trusted_proxies, so Caddy's X-Forwarded-For is honored. The subnet is pinned in searxng.network, so it stays stable across reboots.

Unblocking Google & Search Providers

The current request policy allows a modest amount of VPN latency (4.0s per request, 12.0s maximum) while limiting retries to one. This can recover slow providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429. The default provider policy therefore favors anonymous Bing and DuckDuckGo, disables observed CAPTCHA/429 providers from normal aggregation, and disables pagination to reduce upstream request volume. SearXNG suspends offending engines for hours to days, which is what shows up as "Google went unavailable". This is upstream, not local.

Fixes (cheapest first): tune the outgoing block (request_timeout, retry_on_http_error, suspended_times), duplicate the google engine under a fresh name so a CAPTCHA ban on one leaves the other returning results, and — if Google still re-blocks — route the Google engines at a clean-IP proxy via per-engine proxies in settings.yml. See SearXNG Providers for the full root cause, the suspend-model table, the engines: config, and the /json verification recipe.

Gotchas

• The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in limiter.toml. The only bypass is pass_ip. Local subagents are covered because their slirp/loopback/LAN sources are in pass_ip.
• Agent-API contract (verified live 2026-08-31): the guarded path (search.franksdank.cloud) answers non-browser API clients with Anubis' JS-challenge page at HTTP 200 — a status-code-only client silently gets HTML where it expected JSON. Check content-type, not just status. There is deliberately no way for an API client through the guard (JS challenge + UA deny-list); the agent path is the direct published port http://192.168.0.143:8081/search?q=…&format=json from trusted sources — same instance, VPN-bound egress via Gluetun, unreachable from outside the LAN by construction (no public route to the LAN IP). No second agent-only instance is needed; a duplicate would double upstream request volume from the same rate-sensitive VPN exit IP for zero benefit.
• The config file is root-owned (uid of the rootless podman namespace). The official path is deploy.sh --service searxng: it copies the repo settings and renders the ${VAR} placeholders via scripts/searxng-render-settings.py from /opt/searxng/.env. A hotfix can still hand-copy with sudo cp, but remember the renderer: an unrendered copy silently serves the literal placeholders (before 2026-09-26 the live secret_key literally was "${SEARXNG_SECRET}" and nobody noticed).
• base_url must match the public address or proxy rewrites break.
• Keep the valkey cache container on the same podman network as searxng.

Kagi (paid API engine)

Added 2026-09-26 (user-ordered). The stock SearXNG image ships a native kagi engine module (searx/engines/kagi.py); it authenticates with Authorization: Bearer <key> against POST https://kagi.com/api/v1/search.

• Enabled: weight: 2 alongside bing/duckduckgo, categories [general, web], shortcut kg, kagi_categ: search.
• Cost control: paging: false — exactly one API credit per aggregated query; timeout: 8.0 so a slow paid query is not wasted (scrapers use 5.0).
• Key handling: the repo file keeps api_key: "${KAGI_API_KEY}"; deploy-remote.sh bakes the real key into the live settings.yml via scripts/searxng-render-settings.py, reading /opt/searxng/.env (dank-owned, mode 600). The key never enters git and never appears in any process argv. SearXNG itself does NOT expand env vars in settings.yml — without the renderer the engine fails setup() and is dropped at load.
• Immune to the VPN-exit-IP problem: Kagi authenticates, so it does not care about the shared Gluetun exit IP reputation (see SearXNG Providers).
• Verified live 2026-09-26: direct API probe through the gluetun netns (200, data.search[] shape), then a live aggregated query returned 29/29 results tagged kagi, zero unresponsive engines.

See Also

• Dankclaw Stack
• Gluetun — the VPN container
• SearXNG Providers — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
• Knowledge base: Gluetun, SearXNG, SearXNG Providers, Hermes