serverAppConfigs/docs/Dankclaw/SearXNG Providers.md

# SearXNG Providers

How Google and other upstream search engines get blocked behind the Gluetun
VPN tunnel, and how to unblock them in SearXNG.

**Host:** Dankclaw — **Docs:** https://docs.searxng.org

SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
engine for an hour to 15 days and silently drops it from results. That is the
"Google (and sometimes a few others) went unavailable" symptom.

This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.

## Root cause (one line)

All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
rate-limit it → CAPTCHA → engine **suspension** → empty results.

## The suspension model (what "unavailable" actually is)

An engine that returns 429 / reCAPTCHA / access-denied raises a named
exception and is *suspended* (removed from result aggregation for a set time).
Suspended engines are the usual cause of "gone":

| Exception | Default suspension | Trigger |
|-----------|--------------------|---------|
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |

The values are `search.suspended_times` (+ `ban_time_on_fail`) in
`settings.yml`. An engine that returns **0 results without an error** is *not*
suspended — that is the separate "selector drift" problem (below).

## Two failure modes

| Mode | Symptom | Root cause | Fix family |
|------|---------|-----------|-----------|
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |

Most "Google is broken on a VPN host" cases are a **mix of both**.

## Fix A — Engine config (settings.yml `engines:`)

Important: SearXNG's engines are configured in the **`engines:` list inside
`settings.yml`**, not a separate `engines.yml` file. With
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
the built-in engines by `name` — you do **not** set `use_default_settings: false`.

To add, override, disable, or prune:

```yaml
# shared/searxng/settings.yml  (appended to the existing file)
use_default_settings: true
engines:
  # prune a broken engine entirely
  remove:
    - # (example) wikipedia   # drop an engine that never recovers

  # override + duplicate the Google engines so one survives the other's suspension
  - name: google
    weight: 3
    disabled: false
    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
    retry_on_http_error: [403, 429]
    display_error_messages: true

  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
    engine: google
    categories: general
    weight: 2
    timeout: 5.0
    disabled: false
```

Key per-engine keys (each overwrites the engine-module default; see
`settings_engines.html`):

| Key | Effect |
|-----|--------|
| `name` | merge key (lowercase, no underscore) |
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
| `categories` | categories it serves (also a comma string) |
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
| `weight` | ranks results higher |
| `disabled` | off by default, selectable in /preferences |
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
| `display_error_messages` | show/hide the engine's error banner |
| `proxies` | per-engine egress override (see Fix C) |
| `language` | force region, e.g. `en` |

**Best duplicate trick:** re-declare `google` under a new `name` in a
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.

**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
robust against selector drift, either update the upstream `google.py`
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
Verify `parsers` consumption on the pinned version before depending on it.

**JSON-feed fallback:** decouple parsing from Google's HTML churn.

```yaml
engines:
  - name: google-json
    engine: json_engine
    categories: general
    paging: true
    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
    results_query: results
    title_query: title
    url_query: link
    content_query: snippet
```

## Fix B — Request tuning (settings.yml `outgoing:`)

SearXNG already rotates a **random browser User-Agent per outgoing request**
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
key**. The blogs that point at a `useragents:` table describe the old
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.

Tune the `outgoing` block and the suspend model to survive load:

```yaml
outgoing:
  request_timeout: 6.0             # raise from 2.0 so slow engines aren't dropped as unresponsive
  max_request_timeout: 15.0
  pool_connections: 200            # bigger pool = more concurrent connections per engine host
  pool_maxsize: 40                 # keep-alive connections per host
  enable_http2: true               # multiplexing = fewer distinct "new connection" events per engine
  retries: 3                       # on 429/5xx, retry against a different proxy + source IP
  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you

search:
  ban_time_on_fail: 10
  max_ban_time_on_fail: 180
  suspended_times:
    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries
```

Notes:
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
  `time_between_requests` sleep field does **not** exist in current SearXNG —
  that is a historical-searx concept. The functional equivalent is the suspend
  schedule above (widen to back off before blocking, narrow to recover faster).
- `request_timeout` interacts with the suspend model: too tight → engines look
  "unresponsive"; too loose → slow pages. Aim per-engine `timeout` at
  *proxy-add-on time + engine latency*, capped by `max_request_timeout`.
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
  engine return zero rows and *look* blocked.

## Fix C — Routing & proxies (change which IP the upstreams exit as)

Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
strategy. The decisive fact for our topology: a proxy that runs **inside the
same Gluetun netns** exits from the *same* single IP, so round-robin among
same-IP proxies **does not** unblock. The proxy must egress from a **different,
clean IP**.

```mermaid
graph LR
    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
```

**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
clean IP:

```yaml
engines:
  - name: google
    proxies:
      http:
        - http://gluetun-upstream:8080
      https:
        - http://gluetun-upstream:8080
        - socks5h://gluetun-upstream:8388
  - name: google news
    engine: google_news
    proxies:
      https:
        - http://gluetun-upstream:8080
```

`using_tor_proxy` follows the same per-engine/global pattern.

**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
Gives a second isolated exit IP and reputation pool.
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
  moves to Gluetun, and **split-brain DNS** (below).

**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
via `SERVER_HOSTNAMES`.

**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
rotating the **shared** Gluetun rotates the exit for *all* attached
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
gateway.

> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
> containers/systemd/<unit>.container` `Environment` to inject names into the
> tunnel's `/etc/hosts`.

## Fix D — Verify an engine is unblocked

Three layers, cheapest first. All queries should hit the path production uses
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).

**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
(otherwise `format=json` → 403), then query a single engine:

```bash
BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'
```

Inspect:
- `content[].engine` — which engine produced each result.
- `unresponsive_engines[]` — `{engine, error_type, suspended}`. `error_type`
  tells you *why* (`SearxEngineTooManyRequests`, `SearxEngineCaptcha`,
  `timeout`, `SearxEngineAccessDenied`); `suspended: true` = auto-banned right
  now.
- `timings[].engine` — per-engine latency.

Full status check (bash + jq):

```bash
BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
  curl -s "$BASE/search?q=test&engines=$e&format=json" \
    | jq -r --arg e "$e" '
        (.unresponsive_engines // [])
        | map(select(.engine==$e))
        | if length==0 then "\($e): OK"
          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done
```

**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.

**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance via
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.

Local self-test CLI (`kvch/searx-checker`):

```bash
python3 checker/checker.py http://192.168.0.143:8081
# google.OK  bing.OK  gitlab......ERROR
```

Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
signal), not a direct host port (which uses the host LAN IP and can be
misleading).

## Recommended path for Dankclaw (ranked)

1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
   exactly which engines are down and why. Confirm the running SearXNG version
   (check whether `parsers`/`rate_limit` exist in the pinned build).
2. **Tighten Fix B** — raise `outgoing.request_timeout`, add `retry_on_http_error:
   [403, 429]` on the Google engines, widen `suspended_times`. Cheapest, no new
   infra.
3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
   other returning results.
4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
   dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
   isolation is needed). Avoid the same-netns-proxy trap.

---

## Sources

- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.

## See Also

- [[SearXNG]] — the stack, routing, limiter, health
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
- [[Service Research Brief]] — the research that produced these pages