serverAppConfigs/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md

SearXNG Providers

How Google and other upstream search engines get blocked behind the Gluetun VPN tunnel, and how to unblock them in SearXNG.

Host: Dankclaw — Docs: https://docs.searxng.org

SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave, Wikipedia, …). When all of them are queried through one VPN exit IP — as on Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines start answering with reCAPTCHA or HTTP 429. SearXNG then suspends the offending engine for an hour to 15 days and silently drops it from results. That is the "Google (and sometimes a few others) went unavailable" symptom.

This page is the fixbook. Read SearXNG for the stack; the sub-pages below are the detail if you need it.

Provider policy

The default aggregation should maximize independent, anonymous coverage without turning one user query into a request storm. Disabled engines remain installed and selectable on demand; inactive is reserved for engines that should disappear completely.

Category	Default baseline	On-demand / deferred
General/web	Bing, DuckDuckGo, Wikipedia	Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting
Technical	GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI	Other technical indexes as needed
Science	arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources	Google Scholar until independently tested
News	Reuters, Bing News, DuckDuckGo News, Wikinews	Google News, Brave News, Startpage News
Images	Openverse, Wikimedia Commons, Bing Images, Unsplash	Google/Brave/Startpage image engines
Videos	YouTube, Vimeo, Dailymotion, Bing Videos	Brave video engines

This policy uses no API credentials and does not identify the operator to a search provider. It is deliberately conservative: a bundled engine is not necessarily healthy, anonymous, or suitable for every default query.

Root cause (one line)

All SearXNG engine traffic shares one ProtonVPN exit IP with no rotation, so Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and rate-limit it → CAPTCHA → engine suspension → empty results.

The suspension model (what "unavailable" actually is)

An engine that returns 429 / reCAPTCHA / access-denied raises a named exception and is suspended (removed from result aggregation for a set time). Suspended engines are the usual cause of "gone":

Exception	Default suspension	Trigger
SearxEngineTooManyRequests	~1 hour	HTTP 429
SearxEngineAccessDenied	~1 day	403 / access denied
SearxEngineCaptcha	~1 day	Google /sorry/ or reCAPTCHA
recaptcha_SearxEngineCaptcha	~7 days	Google reCAPTCHA
cf_SearxEngineCaptcha	~15 days	Cloudflare edge CAPTCHA
ordinary errors (timeout/parse)	5–120 s	ban_time_on_fail/max_ban_time_on_fail

The values are search.suspended_times (+ ban_time_on_fail) in settings.yml. An engine that returns 0 results without an error is not suspended — that is the separate "selector drift" problem (below).

Two failure modes

Mode	Symptom	Root cause	Fix family
CAPTCHA / 429 / access-denied	engine flips unresponsive, then suspended for hours	blocked from the shared VPN exit IP	routing/proxies + request tuning (see below)
Selector drift	engine returns 0 results, no error banner	Google renamed its minified CSS classes the engine's hardcoded XPath looks for	engine config (duplicate engine / JSON feed)

Most "Google is broken on a VPN host" cases are a mix of both.

Fix A — Engine config (settings.yml engines:)

The current tracked policy enables Bing and DuckDuckGo for default general search, gives each weight 2, disables pagination, and allows 5 seconds for the VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained but disabled from default aggregation. This limits fan-out while preserving explicit/on-demand selection.

Important: SearXNG's engines are configured in the engines: list inside settings.yml, not a separate engines.yml file. With use_default_settings: true (our setting) your engines: list is merged over the built-in engines by name — you do not set use_default_settings: false.

To add, override, disable, or prune:

# shared/searxng/settings.yml  (appended to the existing file)
use_default_settings: true
engines:
  # prune a broken engine entirely
  remove:
    - # (example) wikipedia   # drop an engine that never recovers

  # override + duplicate the Google engines so one survives the other's suspension
  - name: google
    weight: 3
    disabled: false
    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
    retry_on_http_error: [403, 429]
    display_error_messages: true

  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
    engine: google
    categories: general
    weight: 2
    timeout: 5.0
    disabled: false

Key per-engine keys (each overwrites the engine-module default; see settings_engines.html):

Key	Effect
name	merge key (lowercase, no underscore)
engine	Python module (google, json_engine, xpath)
categories	categories it serves (also a comma string)
timeout	per-engine HTTP timeout, overrides outgoing.request_timeout
weight	ranks results higher
disabled	off by default, selectable in /preferences
inactive	disabled and removed from the UI — use to fully drop a broken engine
retry_on_http_error	retry on status, e.g. true, 403, [403, 429]
display_error_messages	show/hide the engine's error banner
proxies	per-engine egress override (see Fix C)
language	force region, e.g. en

Best duplicate trick: re-declare google under a new name in a different category. When the primary google gets CAPTCHA'd it suspends, but google2 still returns results, and vice-versa.

Parser caveat (read before fiddling with parsers): the modern google, duckduckgo, and brave engines ship a custom response() with hardcoded XPath and ignore the generic parsers/p_parsers table. So parsers: [scrapy_html] under name: google mostly does nothing. To make Google robust against selector drift, either update the upstream google.py selectors (upgrade SearXNG) or run a json_engine against a JSON feed. Verify parsers consumption on the pinned version before depending on it.

JSON-feed fallback: decouple parsing from Google's HTML churn.

engines:
  - name: google-json
    engine: json_engine
    categories: general
    paging: true
    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
    results_query: results
    title_query: title
    url_query: link
    content_query: snippet

Fix B — Request tuning (settings.yml outgoing:)

SearXNG already rotates a random browser User-Agent per outgoing request (gen_useragent() in online.py) — there is no native useragents: mapping key. The blogs that point at a useragents: table describe the old asciimoo/searx; current SearXNG rotates automatically. Keep relying on it.

Tune the outgoing block and the suspend model to survive load:

outgoing:
  request_timeout: 4.0             # current Dankclaw value; modest VPN-latency allowance
  max_request_timeout: 12.0
  pool_connections: 100
  pool_maxsize: 20
  enable_http2: true
  retries: 1                       # avoid amplifying 429/CAPTCHA responses from one exit IP
  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you

search:
  ban_time_on_fail: 10
  max_ban_time_on_fail: 180
  suspended_times:
    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries

Notes:

• Auto-suspend is the real "rate limit". The old per-engine rate_limit / time_between_requests sleep field does not exist in current SearXNG — that is a historical-searx concept. The functional equivalent is the suspend schedule above (widen to back off before blocking, narrow to recover faster).
• request_timeout interacts with the suspend model: too tight → engines look "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and 12 seconds as the cap; raise it only when measurements show real upstream latency rather than 429/CAPTCHA responses.
• Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the provider's rate limit worse. A retry count of 1 is a bounded compromise for a transient network failure.
• safe_search: 0 when diagnosing — a too-strict global safe search can make an engine return zero rows and look blocked.

Fix C — Routing & proxies (change which IP the upstreams exit as)

Two independent levers: SearXNG's own upstream proxies, and the Gluetun VPN strategy. The decisive fact for our topology: a proxy that runs inside the same Gluetun netns exits from the same single IP, so round-robin among same-IP proxies does not unblock. The proxy must egress from a different, clean IP.

graph LR
    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]

Option 1 — per-engine proxies (cheapest, settings.yml only). Point the heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a clean IP:

engines:
  - name: google
    proxies:
      http:
        - http://gluetun-upstream:8080
      https:
        - http://gluetun-upstream:8080
        - socks5h://gluetun-upstream:8388
  - name: google news
    engine: google_news
    proxies:
      https:
        - http://gluetun-upstream:8080

using_tor_proxy follows the same per-engine/global pattern.

Option 2 — dedicated second Gluetun for SearXNG's upstreams. Run a second Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy (8080, HTTPPROXY=on), and point the Google engines at it. SearXNG inbound (still on the main host/Caddy) stays put; only upstream requests re-route. Gives a second isolated exit IP and reputation pool.

• Cons: a second ProtenVPN device slot, a new container, port publishing moves to Gluetun, and split-brain DNS (below).

Option 3 — dedicated ProtonVPN IP. A static, exclusive exit IP that other subscribers can't pollute. Requires a ProtonVPN Business plan as of the sources reviewed (the long-requested personal-plan dedicated IP is still only on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node via SERVER_HOSTNAMES.

Option 4 — rotate the exit. List SERVER_COUNTRIES/SERVER_CITIES/ SERVER_HOSTNAMES so Gluetun picks from several physical exit servers. Caveat: rotating the shared Gluetun rotates the exit for all attached containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated gateway.

⚠️ Split-brain DNS: a container sharing Gluetun's netns (Network=gluetun.container) loses Docker's internal service-name DNS — it can't resolve siblings by name. Use a static IP or extra_hosts/~/.config/ containers/systemd/<unit>.container Environment to inject names into the tunnel's /etc/hosts.

Observed Dankclaw baseline (2026-08-25)

The read-only audit found:

• SearXNG image: ghcr.io/searxng/searxng:latest; embedded version 1.0.0.
• Gluetun uses VPN_SERVICE_PROVIDER=custom with a static WireGuard profile; no SERVER_COUNTRIES, SERVER_CITIES, SERVER_REGIONS, or SERVER_HOSTNAMES filters are configured.
• The observed exit was 149.22.88.19 (Seattle, Washington, US).
• Bing and DuckDuckGo returned results. Mojeek returned no results without an upstream error.
• google cse and Brave were suspended for too many requests; Startpage was suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded Google's "unusual traffic from your network" response.
• /healthz returned 200 OK; the SearXNG service and Gluetun service were healthy. This isolates the primary failures to upstream provider handling, not application availability.

This deployment returns the legacy array form for unresponsive_engines:

"unresponsive_engines": [
  ["brave", "Suspended: too many requests"],
  ["startpage", "Suspended: CAPTCHA"]
]

Do not assume the newer object form (engine, error_type, suspended) until the live response has been checked. The direct published port is the useful diagnostic path because it exercises the ProtonVPN exit. The Caddy hostname resolved to the WAN address during the audit and timed out from the audit host, so it was not used as the baseline path.

Fix D — Verify an engine is unblocked

Three layers, cheapest first. All queries should hit the path production uses (the published Gluetun port → ProtonVPN exit, not a direct host port).

1. Isolate one engine with /json. Enable json in search.formats (otherwise format=json → 403), then query a single engine:

BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'

Inspect:

• content[].engine — which engine produced each result.
• unresponsive_engines[] — inspect the live shape first. On Dankclaw it is [engine, message], for example ["google cse", "Suspended: too many requests"]. Newer builds may return objects containing engine, error_type, and suspended.
• timings[].engine — per-engine latency when the running build emits timings.

Full status check (bash + jq):

BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
  curl -s "$BASE/search?q=test&engines=$e&format=json" \
    | jq -r --arg e "$e" '
        (.unresponsive_engines // [])
        | map(select(.engine==$e))
        | if length==0 then "\($e): OK"
          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done

2. /config REST endpoint lists every engine + enabled/disabled + categories — an inventory of what is live. ?engines=<name> isolates a single provider; /debug adds engine diagnostics.

3. check.searx.space — the hosted official checker probes every engine of your instance with a headless Firefox (catches Cloudflare/CAPTCHA blocks a plain HTTP probe misses) and posts results on searx.space. Add your instance via a GitHub issue in searxng/searx-instances, then allow check.searx.space IPs to reach the SearXNG port.

Local self-test CLI (kvch/searx-checker):

python3 checker/checker.py http://192.168.0.143:8081
# google.OK  bing.OK  gitlab......ERROR

Verify through the published Gluetun port (realistic ProtonVPN-exit block signal), not a direct host port (which uses the host LAN IP and can be misleading).

Recommended path for Dankclaw (ranked)

1. Measure individual engines — use /json + ?engines=<name> and several query classes before enabling a bundled engine in default aggregation.
2. Keep default fan-out small — prefer independent anonymous providers and disable pagination for rate-sensitive engines.
3. Promote by evidence — raise weight only for engines that return useful results consistently; weight is not a health check.
4. Avoid credentials — do not add API-backed engines unless the provider is genuinely free, anonymous, and compatible with our privacy requirements.
5. Verify first — /json + ?engines=google and the searx-checker to see exactly which engines are down and why. Confirm the running SearXNG version (check whether parsers/rate_limit exist in the pinned build).
6. Use conservative request tuning — Dankclaw now allows 5 seconds per upstream request, caps it at 15 seconds, and retries once. This addresses modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only after measuring the resulting error and latency profile.
7. If Google still re-blocks — Fix C Option 1: point the Google engines at a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation isolation is needed). Avoid the same-netns-proxy trap.

---

Sources

• SearXNG settings: engines — per-engine keys.
• SearXNG settings: outgoing — request_timeout, pool, retries, proxies, source_ips.
• SearXNG settings: search — suspended_times, ban_time_on_fail.
• SearXNG settings (merge + remove/keep_only) — use_default_settings semantics.
• Google engine docs — detect_google_sorry, WML/Nokia XML, max_page.
• json_engine — json parser config.
• SearXNG exceptions — exception types and suspend settings.
• Google engine stops yielding results #5286 — reproducible "Google unavailable".
• DuckDuckGo CAPTCHA #4824 — per-engine CAPTCHA signature.
• Google engine actively blocking IPs #2515
• answer-captcha from server's IP — block is tied to the egress IP.
• SearXNG rate limits / 429 (SSD Nodes) — suspension table, /stats/errors.
• How search engines detect scrapers (Crawlbase) — IP reputation + rate + CAPTCHA mechanics.
• check.searx.space + kvch/searx-checker — the checkers.
• searx-instances — how to be checked — add your instance + allow checker IPs.
• ProtonVPN — Dedicated IP addresses + Benefits — IP-reputation reasoning; Business plan (flag: personal dedicated IP still on uservoice).
• Gluetun wiki — ProtonVPN provider — SERVER_HOSTNAMES narrowest-filter risk.
• Gluetun wiki — Built-in Proxies — HTTP 8080 + Shadowsocks 8388.
• Gluetun wiki — Container networking — netns sharing, port publishing on Gluetun.
• Gluetun inter-network DNS failures (Jacob Jangles) — split-brain DNS + fix.

See Also

• SearXNG — the stack, routing, limiter, health
• Gluetun — the VPN tunnel (where the exit IP comes from)
• Dankclaw Stack — the service the SearXNG stack sits in
• Service Research Brief — the research that produced these pages