docs: add SearXNG providers guide for unblocking Google behind Gluetun VPN

SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP, so
rate-sensitive engines (Google, DuckDuckGo) answer with reCAPTCHA/429 and
SearXNG suspends the offending engine for hours-to-days, showing up as
"unavailable". Add docs/Dankclaw/SearXNG Providers.md covering: the
suspension model, the two failure modes (CAPTCHA vs selector drift), engine
config (settings.yml engines: list — duplicate google, timeout,
retry_on_http_error, JSON feed), request tuning (outgoing block, UA rotation,
suspended_times — correcting the old rate_limit/useragents blog myths),
routing/proxies (per-engine proxies, dedicated 2nd Gluetun, dedicated IP), and
verification (/json, /config, check.searx.space). Crosslink into SearXNG.md
and index.md.
This commit is contained in:
Frank committed 2026-08-25 06:22:57 -07:00
1 parent b457457bfe
commit 24efa98d56
3 files changed
+349 -2

No files matched your search

+18 -2
View File
@@ -94,7 +94,22 @@ The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The const
Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
## Gotchas
## Unblocking Google & Search Providers
SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The
rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
and SearXNG **suspends** the offending engine for hours to days, which is what
shows up as "Google went unavailable". This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
if Google still re-blocks — route the Google engines at a clean-IP proxy via
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
root cause, the suspend-model table, the `engines:` config, and the
`/json` verification recipe.
### Gotchas
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
- The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
@@ -106,4 +121,5 @@ Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local s
- [[Dankclaw Stack]]
- [[Gluetun]] — the VPN container
- Knowledge base: [[Gluetun]], [[SearXNG]], [[Hermes]]
- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]