dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add SearXNG providers guide for unblocking Google behind Gluetun VPN

Browse Source 
SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP, so
rate-sensitive engines (Google, DuckDuckGo) answer with reCAPTCHA/429 and
SearXNG suspends the offending engine for hours-to-days, showing up as
"unavailable". Add docs/Dankclaw/SearXNG Providers.md covering: the
suspension model, the two failure modes (CAPTCHA vs selector drift), engine
config (settings.yml engines: list — duplicate google, timeout,
retry_on_http_error, JSON feed), request tuning (outgoing block, UA rotation,
suspended_times — correcting the old rate_limit/useragents blog myths),
routing/proxies (per-engine proxies, dedicated 2nd Gluetun, dedicated IP), and
verification (/json, /config, check.searx.space). Crosslink into SearXNG.md
and index.md.
This commit is contained in:
Frank
2026-08-25 06:22:57 -07:00
parent b457457bfe
commit 24efa98d56
3 changed files with 349 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Dankclaw/SearXNG Providers.md
+330
View File
@@ -0,0 +1,330 @@
# SearXNG Providers
How Google and other upstream search engines get blocked behind the Gluetun
VPN tunnel, and how to unblock them in SearXNG.
**Host:** Dankclaw — **Docs:** https://docs.searxng.org
SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
engine for an hour to 15 days and silently drops it from results. That is the
"Google (and sometimes a few others) went unavailable" symptom.
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.
## Root cause (one line)
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
rate-limit it → CAPTCHA → engine **suspension** → empty results.
## The suspension model (what "unavailable" actually is)
An engine that returns 429 / reCAPTCHA / access-denied raises a named
exception and is *suspended* (removed from result aggregation for a set time).
Suspended engines are the usual cause of "gone":
| Exception | Default suspension | Trigger |
|-----------|--------------------|---------|
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
The values are `search.suspended_times` (+ `ban_time_on_fail`) in
`settings.yml`. An engine that returns **0 results without an error** is *not*
suspended — that is the separate "selector drift" problem (below).
## Two failure modes
| Mode | Symptom | Root cause | Fix family |
|------|---------|-----------|-----------|
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |
Most "Google is broken on a VPN host" cases are a **mix of both**.
## Fix A — Engine config (settings.yml `engines:`)
Important: SearXNG's engines are configured in the **`engines:` list inside
`settings.yml`**, not a separate `engines.yml` file. With
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
the built-in engines by `name` — you do **not** set `use_default_settings: false`.
To add, override, disable, or prune:
```yaml
# shared/searxng/settings.yml (appended to the existing file)
use_default_settings: true
engines:
# prune a broken engine entirely
remove:
- # (example) wikipedia # drop an engine that never recovers
# override + duplicate the Google engines so one survives the other's suspension
- name: google
weight: 3
disabled: false
timeout: 5.0 # wait longer over the tunnel (global default is 2.0s)
retry_on_http_error: [403, 429]
display_error_messages: true
- name: google2 # fresh engine, same module, survives google's CAPTCHA ban
engine: google
categories: general
weight: 2
timeout: 5.0
disabled: false
```
Key per-engine keys (each overwrites the engine-module default; see
`settings_engines.html`):
| Key | Effect |
|-----|--------|
| `name` | merge key (lowercase, no underscore) |
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
| `categories` | categories it serves (also a comma string) |
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
| `weight` | ranks results higher |
| `disabled` | off by default, selectable in /preferences |
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
| `display_error_messages` | show/hide the engine's error banner |
| `proxies` | per-engine egress override (see Fix C) |
| `language` | force region, e.g. `en` |
**Best duplicate trick:** re-declare `google` under a new `name` in a
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.
**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
robust against selector drift, either update the upstream `google.py`
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
Verify `parsers` consumption on the pinned version before depending on it.
**JSON-feed fallback:** decouple parsing from Google's HTML churn.
```yaml
engines:
- name: google-json
engine: json_engine
categories: general
paging: true
search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
results_query: results
title_query: title
url_query: link
content_query: snippet
```
## Fix B — Request tuning (settings.yml `outgoing:`)
SearXNG already rotates a **random browser User-Agent per outgoing request**
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
key**. The blogs that point at a `useragents:` table describe the old
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.
Tune the `outgoing` block and the suspend model to survive load:
```yaml
outgoing:
request_timeout: 6.0 # raise from 2.0 so slow engines aren't dropped as unresponsive
max_request_timeout: 15.0
pool_connections: 200 # bigger pool = more concurrent connections per engine host
pool_maxsize: 40 # keep-alive connections per host
enable_http2: true # multiplexing = fewer distinct "new connection" events per engine
retries: 3 # on 429/5xx, retry against a different proxy + source IP
useragent_suffix: "ops@yourhomelab.example" # so a per-engine blocker can reach you
search:
ban_time_on_fail: 10
max_ban_time_on_fail: 180
suspended_times:
SearxEngineTooManyRequests: 300 # 429 → 5 min before it retries (tune down to recover faster)
SearxEngineCaptcha: 7200 # CAPTCHA → 2h before it retries
```
Notes:
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
`time_between_requests` sleep field does **not** exist in current SearXNG —
that is a historical-searx concept. The functional equivalent is the suspend
schedule above (widen to back off before blocking, narrow to recover faster).
- `request_timeout` interacts with the suspend model: too tight → engines look
"unresponsive"; too loose → slow pages. Aim per-engine `timeout` at
*proxy-add-on time + engine latency*, capped by `max_request_timeout`.
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
engine return zero rows and *look* blocked.
## Fix C — Routing & proxies (change which IP the upstreams exit as)
Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
strategy. The decisive fact for our topology: a proxy that runs **inside the
same Gluetun netns** exits from the *same* single IP, so round-robin among
same-IP proxies **does not** unblock. The proxy must egress from a **different,
clean IP**.
```mermaid
graph LR
SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
```
**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
clean IP:
```yaml
engines:
- name: google
proxies:
http:
- http://gluetun-upstream:8080
https:
- http://gluetun-upstream:8080
- socks5h://gluetun-upstream:8388
- name: google news
engine: google_news
proxies:
https:
- http://gluetun-upstream:8080
```
`using_tor_proxy` follows the same per-engine/global pattern.
**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
Gives a second isolated exit IP and reputation pool.
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
moves to Gluetun, and **split-brain DNS** (below).
**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
via `SERVER_HOSTNAMES`.
**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
rotating the **shared** Gluetun rotates the exit for *all* attached
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
gateway.
> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
> containers/systemd/<unit>.container` `Environment` to inject names into the
> tunnel's `/etc/hosts`.
## Fix D — Verify an engine is unblocked
Three layers, cheapest first. All queries should hit the path production uses
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
(otherwise `format=json` → 403), then query a single engine:
```bash
BASE="http://192.168.0.143:8081" # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
| jq -r '.unresponsive_engines[]? | select(.engine=="google")'
```
Inspect:
- `content[].engine` — which engine produced each result.
- `unresponsive_engines[]` — `{engine, error_type, suspended}`. `error_type`
tells you *why* (`SearxEngineTooManyRequests`, `SearxEngineCaptcha`,
`timeout`, `SearxEngineAccessDenied`); `suspended: true` = auto-banned right
now.
- `timings[].engine` — per-engine latency.
Full status check (bash + jq):
```bash
BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
curl -s "$BASE/search?q=test&engines=$e&format=json" \
| jq -r --arg e "$e" '
(.unresponsive_engines // [])
| map(select(.engine==$e))
| if length==0 then "\($e): OK"
else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done
```
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.
**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance via
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.
Local self-test CLI (`kvch/searx-checker`):
```bash
python3 checker/checker.py http://192.168.0.143:8081
# google.OK bing.OK gitlab......ERROR
```
Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
signal), not a direct host port (which uses the host LAN IP and can be
misleading).
## Recommended path for Dankclaw (ranked)
1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
2. **Tighten Fix B** — raise `outgoing.request_timeout`, add `retry_on_http_error:
[403, 429]` on the Google engines, widen `suspended_times`. Cheapest, no new
infra.
3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
other returning results.
4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
isolation is needed). Avoid the same-netns-proxy trap.
---
## Sources
- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
## See Also
- [[SearXNG]] — the stack, routing, limiter, health
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
- [[Service Research Brief]] — the research that produced these pages
docs/Dankclaw/SearXNG.md
+18 -2
View File
@@ -94,7 +94,22 @@ The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The const
Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
## Gotchas
## Unblocking Google & Search Providers
SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The
rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
and SearXNG **suspends** the offending engine for hours to days, which is what
shows up as "Google went unavailable". This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
if Google still re-blocks — route the Google engines at a clean-IP proxy via
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
root cause, the suspend-model table, the `engines:` config, and the
`/json` verification recipe.
### Gotchas
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
- The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
@@ -106,4 +121,5 @@ Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local s
- [[Dankclaw Stack]]
- [[Gluetun]] — the VPN container
- Knowledge base: [[Gluetun]], [[SearXNG]], [[Hermes]]
- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]
docs/index.md
+1
View File
@@ -51,6 +51,7 @@ covers both workloads).
## Dankclaw Services
- [[SearXNG]] — privacy metasearch (VPN-routed)
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
- [[Gluetun]] — the VPN tunnel container
- [[Valkey]] — SearXNG cache and limiter store
- [[ComfyUI]] — node-based AI image generation (CPU)