serverAppConfigs/docs/Dankclaw/SearXNG Providers.md

SearXNG Providers

How Google and other upstream search engines get blocked behind the Gluetun VPN tunnel, and how to unblock them in SearXNG.

Host: Dankclaw — Docs: https://docs.searxng.org

SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave, Wikipedia, …). When all of them are queried through one VPN exit IP — as on Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines start answering with reCAPTCHA or HTTP 429. SearXNG then suspends the offending engine for an hour to 15 days and silently drops it from results. That is the "Google (and sometimes a few others) went unavailable" symptom.

This page is the fixbook. Read SearXNG for the stack; the sub-pages below are the detail if you need it.

Root cause (one line)

All SearXNG engine traffic shares one ProtonVPN exit IP with no rotation, so Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and rate-limit it → CAPTCHA → engine suspension → empty results.

The suspension model (what "unavailable" actually is)

An engine that returns 429 / reCAPTCHA / access-denied raises a named exception and is suspended (removed from result aggregation for a set time). Suspended engines are the usual cause of "gone":

Exception	Default suspension	Trigger
SearxEngineTooManyRequests	~1 hour	HTTP 429
SearxEngineAccessDenied	~1 day	403 / access denied
SearxEngineCaptcha	~1 day	Google /sorry/ or reCAPTCHA
recaptcha_SearxEngineCaptcha	~7 days	Google reCAPTCHA
cf_SearxEngineCaptcha	~15 days	Cloudflare edge CAPTCHA
ordinary errors (timeout/parse)	5–120 s	ban_time_on_fail/max_ban_time_on_fail

The values are search.suspended_times (+ ban_time_on_fail) in settings.yml. An engine that returns 0 results without an error is not suspended — that is the separate "selector drift" problem (below).

Two failure modes

Mode	Symptom	Root cause	Fix family
CAPTCHA / 429 / access-denied	engine flips unresponsive, then suspended for hours	blocked from the shared VPN exit IP	routing/proxies + request tuning (see below)
Selector drift	engine returns 0 results, no error banner	Google renamed its minified CSS classes the engine's hardcoded XPath looks for	engine config (duplicate engine / JSON feed)

Most "Google is broken on a VPN host" cases are a mix of both.

Fix A — Engine config (settings.yml engines:)

Important: SearXNG's engines are configured in the engines: list inside settings.yml, not a separate engines.yml file. With use_default_settings: true (our setting) your engines: list is merged over the built-in engines by name — you do not set use_default_settings: false.

To add, override, disable, or prune:

# shared/searxng/settings.yml  (appended to the existing file)
use_default_settings: true
engines:
  # prune a broken engine entirely
  remove:
    - # (example) wikipedia   # drop an engine that never recovers

  # override + duplicate the Google engines so one survives the other's suspension
  - name: google
    weight: 3
    disabled: false
    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
    retry_on_http_error: [403, 429]
    display_error_messages: true

  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
    engine: google
    categories: general
    weight: 2
    timeout: 5.0
    disabled: false

Key per-engine keys (each overwrites the engine-module default; see settings_engines.html):

Key	Effect
name	merge key (lowercase, no underscore)
engine	Python module (google, json_engine, xpath)
categories	categories it serves (also a comma string)
timeout	per-engine HTTP timeout, overrides outgoing.request_timeout
weight	ranks results higher
disabled	off by default, selectable in /preferences
inactive	disabled and removed from the UI — use to fully drop a broken engine
retry_on_http_error	retry on status, e.g. true, 403, [403, 429]
display_error_messages	show/hide the engine's error banner
proxies	per-engine egress override (see Fix C)
language	force region, e.g. en

Best duplicate trick: re-declare google under a new name in a different category. When the primary google gets CAPTCHA'd it suspends, but google2 still returns results, and vice-versa.

Parser caveat (read before fiddling with parsers): the modern google, duckduckgo, and brave engines ship a custom response() with hardcoded XPath and ignore the generic parsers/p_parsers table. So parsers: [scrapy_html] under name: google mostly does nothing. To make Google robust against selector drift, either update the upstream google.py selectors (upgrade SearXNG) or run a json_engine against a JSON feed. Verify parsers consumption on the pinned version before depending on it.

JSON-feed fallback: decouple parsing from Google's HTML churn.

engines:
  - name: google-json
    engine: json_engine
    categories: general
    paging: true
    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
    results_query: results
    title_query: title
    url_query: link
    content_query: snippet

Fix B — Request tuning (settings.yml outgoing:)

SearXNG already rotates a random browser User-Agent per outgoing request (gen_useragent() in online.py) — there is no native useragents: mapping key. The blogs that point at a useragents: table describe the old asciimoo/searx; current SearXNG rotates automatically. Keep relying on it.

Tune the outgoing block and the suspend model to survive load:

outgoing:
  request_timeout: 6.0             # raise from 2.0 so slow engines aren't dropped as unresponsive
  max_request_timeout: 15.0
  pool_connections: 200            # bigger pool = more concurrent connections per engine host
  pool_maxsize: 40                 # keep-alive connections per host
  enable_http2: true               # multiplexing = fewer distinct "new connection" events per engine
  retries: 3                       # on 429/5xx, retry against a different proxy + source IP
  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you

search:
  ban_time_on_fail: 10
  max_ban_time_on_fail: 180
  suspended_times:
    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries

Notes:

• Auto-suspend is the real "rate limit". The old per-engine rate_limit / time_between_requests sleep field does not exist in current SearXNG — that is a historical-searx concept. The functional equivalent is the suspend schedule above (widen to back off before blocking, narrow to recover faster).
• request_timeout interacts with the suspend model: too tight → engines look "unresponsive"; too loose → slow pages. Aim per-engine timeout at proxy-add-on time + engine latency, capped by max_request_timeout.
• safe_search: 0 when diagnosing — a too-strict global safe search can make an engine return zero rows and look blocked.

Fix C — Routing & proxies (change which IP the upstreams exit as)

Two independent levers: SearXNG's own upstream proxies, and the Gluetun VPN strategy. The decisive fact for our topology: a proxy that runs inside the same Gluetun netns exits from the same single IP, so round-robin among same-IP proxies does not unblock. The proxy must egress from a different, clean IP.

graph LR
    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]

Option 1 — per-engine proxies (cheapest, settings.yml only). Point the heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a clean IP:

engines:
  - name: google
    proxies:
      http:
        - http://gluetun-upstream:8080
      https:
        - http://gluetun-upstream:8080
        - socks5h://gluetun-upstream:8388
  - name: google news
    engine: google_news
    proxies:
      https:
        - http://gluetun-upstream:8080

using_tor_proxy follows the same per-engine/global pattern.

Option 2 — dedicated second Gluetun for SearXNG's upstreams. Run a second Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy (8080, HTTPPROXY=on), and point the Google engines at it. SearXNG inbound (still on the main host/Caddy) stays put; only upstream requests re-route. Gives a second isolated exit IP and reputation pool.

• Cons: a second ProtenVPN device slot, a new container, port publishing moves to Gluetun, and split-brain DNS (below).

Option 3 — dedicated ProtonVPN IP. A static, exclusive exit IP that other subscribers can't pollute. Requires a ProtonVPN Business plan as of the sources reviewed (the long-requested personal-plan dedicated IP is still only on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node via SERVER_HOSTNAMES.

Option 4 — rotate the exit. List SERVER_COUNTRIES/SERVER_CITIES/ SERVER_HOSTNAMES so Gluetun picks from several physical exit servers. Caveat: rotating the shared Gluetun rotates the exit for all attached containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated gateway.

⚠️ Split-brain DNS: a container sharing Gluetun's netns (Network=gluetun.container) loses Docker's internal service-name DNS — it can't resolve siblings by name. Use a static IP or extra_hosts/~/.config/ containers/systemd/<unit>.container Environment to inject names into the tunnel's /etc/hosts.

Fix D — Verify an engine is unblocked

Three layers, cheapest first. All queries should hit the path production uses (the published Gluetun port → ProtonVPN exit, not a direct host port).

1. Isolate one engine with /json. Enable json in search.formats (otherwise format=json → 403), then query a single engine:

BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'

Inspect:

• content[].engine — which engine produced each result.
• unresponsive_engines[] — {engine, error_type, suspended}. error_type tells you why (SearxEngineTooManyRequests, SearxEngineCaptcha, timeout, SearxEngineAccessDenied); suspended: true = auto-banned right now.
• timings[].engine — per-engine latency.

Full status check (bash + jq):

BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
  curl -s "$BASE/search?q=test&engines=$e&format=json" \
    | jq -r --arg e "$e" '
        (.unresponsive_engines // [])
        | map(select(.engine==$e))
        | if length==0 then "\($e): OK"
          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done

2. /config REST endpoint lists every engine + enabled/disabled + categories — an inventory of what is live. ?engines=<name> isolates a single provider; /debug adds engine diagnostics.

3. check.searx.space — the hosted official checker probes every engine of your instance with a headless Firefox (catches Cloudflare/CAPTCHA blocks a plain HTTP probe misses) and posts results on searx.space. Add your instance via a GitHub issue in searxng/searx-instances, then allow check.searx.space IPs to reach the SearXNG port.

Local self-test CLI (kvch/searx-checker):

python3 checker/checker.py http://192.168.0.143:8081
# google.OK  bing.OK  gitlab......ERROR

Verify through the published Gluetun port (realistic ProtonVPN-exit block signal), not a direct host port (which uses the host LAN IP and can be misleading).

Recommended path for Dankclaw (ranked)

1. Verify first — /json + ?engines=google and the searx-checker to see exactly which engines are down and why. Confirm the running SearXNG version (check whether parsers/rate_limit exist in the pinned build).
2. Tighten Fix B — raise outgoing.request_timeout, add retry_on_http_error: [403, 429] on the Google engines, widen suspended_times. Cheapest, no new infra.
3. Duplicate the Google engines (Fix A) so a CAPTCHA ban on one leaves the other returning results.
4. If Google still re-blocks — Fix C Option 1: point the Google engines at a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation isolation is needed). Avoid the same-netns-proxy trap.

---

Sources

• SearXNG settings: engines — per-engine keys.
• SearXNG settings: outgoing — request_timeout, pool, retries, proxies, source_ips.
• SearXNG settings: search — suspended_times, ban_time_on_fail.
• SearXNG settings (merge + remove/keep_only) — use_default_settings semantics.
• Google engine docs — detect_google_sorry, WML/Nokia XML, max_page.
• json_engine — json parser config.
• SearXNG exceptions — exception types and suspend settings.
• Google engine stops yielding results #5286 — reproducible "Google unavailable".
• DuckDuckGo CAPTCHA #4824 — per-engine CAPTCHA signature.
• Google engine actively blocking IPs #2515
• answer-captcha from server's IP — block is tied to the egress IP.
• SearXNG rate limits / 429 (SSD Nodes) — suspension table, /stats/errors.
• How search engines detect scrapers (Crawlbase) — IP reputation + rate + CAPTCHA mechanics.
• check.searx.space + kvch/searx-checker — the checkers.
• searx-instances — how to be checked — add your instance + allow checker IPs.
• ProtonVPN — Dedicated IP addresses + Benefits — IP-reputation reasoning; Business plan (flag: personal dedicated IP still on uservoice).
• Gluetun wiki — ProtonVPN provider — SERVER_HOSTNAMES narrowest-filter risk.
• Gluetun wiki — Built-in Proxies — HTTP 8080 + Shadowsocks 8388.
• Gluetun wiki — Container networking — netns sharing, port publishing on Gluetun.
• Gluetun inter-network DNS failures (Jacob Jangles) — split-brain DNS + fix.

See Also

• SearXNG — the stack, routing, limiter, health
• Gluetun — the VPN tunnel (where the exit IP comes from)
• Dankclaw Stack — the service the SearXNG stack sits in
• Service Research Brief — the research that produced these pages