serverAppConfigs/docs/Dankclaw/SearXNG Providers.md

# SearXNG Providers

How Google and other upstream search engines get blocked behind the Gluetun
VPN tunnel, and how to unblock them in SearXNG.

**Host:** Dankclaw — **Docs:** https://docs.searxng.org

SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
engine for an hour to 15 days and silently drops it from results. That is the
"Google (and sometimes a few others) went unavailable" symptom.

This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.

## Provider policy

The default aggregation should maximize **independent, anonymous coverage**
without turning one user query into a request storm. Disabled engines remain
installed and selectable on demand; `inactive` is reserved for engines that
should disappear completely.

| Category | Default baseline | On-demand / deferred |
|----------|------------------|----------------------|
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |

This policy uses no API credentials and does not identify the operator to a
search provider. It is deliberately conservative: a bundled engine is not
necessarily healthy, anonymous, or suitable for every default query.

## Root cause (one line)

All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
rate-limit it → CAPTCHA → engine **suspension** → empty results.

## The suspension model (what "unavailable" actually is)

An engine that returns 429 / reCAPTCHA / access-denied raises a named
exception and is *suspended* (removed from result aggregation for a set time).
Suspended engines are the usual cause of "gone":

| Exception | Default suspension | Trigger |
|-----------|--------------------|---------|
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |

The values are `search.suspended_times` (+ `ban_time_on_fail`) in
`settings.yml`. An engine that returns **0 results without an error** is *not*
suspended — that is the separate "selector drift" problem (below).

## Two failure modes

| Mode | Symptom | Root cause | Fix family |
|------|---------|-----------|-----------|
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |

Most "Google is broken on a VPN host" cases are a **mix of both**.

## Fix A — Engine config (settings.yml `engines:`)

The current tracked policy enables Bing and DuckDuckGo for default general
search, gives each weight 2, disables pagination, and allows 5 seconds for the
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
but disabled from default aggregation. This limits fan-out while preserving
explicit/on-demand selection.

Important: SearXNG's engines are configured in the **`engines:` list inside
`settings.yml`**, not a separate `engines.yml` file. With
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
the built-in engines by `name` — you do **not** set `use_default_settings: false`.

To add, override, disable, or prune:

```yaml
# shared/searxng/settings.yml  (appended to the existing file)
use_default_settings: true
engines:
  # prune a broken engine entirely
  remove:
    - # (example) wikipedia   # drop an engine that never recovers

  # override + duplicate the Google engines so one survives the other's suspension
  - name: google
    weight: 3
    disabled: false
    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
    retry_on_http_error: [403, 429]
    display_error_messages: true

  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
    engine: google
    categories: general
    weight: 2
    timeout: 5.0
    disabled: false
```

Key per-engine keys (each overwrites the engine-module default; see
`settings_engines.html`):

| Key | Effect |
|-----|--------|
| `name` | merge key (lowercase, no underscore) |
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
| `categories` | categories it serves (also a comma string) |
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
| `weight` | ranks results higher |
| `disabled` | off by default, selectable in /preferences |
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
| `display_error_messages` | show/hide the engine's error banner |
| `proxies` | per-engine egress override (see Fix C) |
| `language` | force region, e.g. `en` |

**Best duplicate trick:** re-declare `google` under a new `name` in a
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.

**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
robust against selector drift, either update the upstream `google.py`
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
Verify `parsers` consumption on the pinned version before depending on it.

**JSON-feed fallback:** decouple parsing from Google's HTML churn.

```yaml
engines:
  - name: google-json
    engine: json_engine
    categories: general
    paging: true
    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
    results_query: results
    title_query: title
    url_query: link
    content_query: snippet
```

## Fix B — Request tuning (settings.yml `outgoing:`)

SearXNG already rotates a **random browser User-Agent per outgoing request**
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
key**. The blogs that point at a `useragents:` table describe the old
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.

Tune the `outgoing` block and the suspend model to survive load:

```yaml
outgoing:
  request_timeout: 4.0             # current Dankclaw value; modest VPN-latency allowance
  max_request_timeout: 12.0
  pool_connections: 100
  pool_maxsize: 20
  enable_http2: true
  retries: 1                       # avoid amplifying 429/CAPTCHA responses from one exit IP
  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you

search:
  ban_time_on_fail: 10
  max_ban_time_on_fail: 180
  suspended_times:
    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries
```

Notes:
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
  `time_between_requests` sleep field does **not** exist in current SearXNG —
  that is a historical-searx concept. The functional equivalent is the suspend
  schedule above (widen to back off before blocking, narrow to recover faster).
- `request_timeout` interacts with the suspend model: too tight → engines look
  "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
  12 seconds as the cap; raise it only when measurements show real upstream
  latency rather than 429/CAPTCHA responses.
- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
  provider's rate limit worse. A retry count of 1 is a bounded compromise for a
  transient network failure.
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
  engine return zero rows and *look* blocked.

## Fix C — Routing & proxies (change which IP the upstreams exit as)

Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
strategy. The decisive fact for our topology: a proxy that runs **inside the
same Gluetun netns** exits from the *same* single IP, so round-robin among
same-IP proxies **does not** unblock. The proxy must egress from a **different,
clean IP**.

```mermaid
graph LR
    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
```

**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
clean IP:

```yaml
engines:
  - name: google
    proxies:
      http:
        - http://gluetun-upstream:8080
      https:
        - http://gluetun-upstream:8080
        - socks5h://gluetun-upstream:8388
  - name: google news
    engine: google_news
    proxies:
      https:
        - http://gluetun-upstream:8080
```

`using_tor_proxy` follows the same per-engine/global pattern.

**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
Gives a second isolated exit IP and reputation pool.
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
  moves to Gluetun, and **split-brain DNS** (below).

**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
via `SERVER_HOSTNAMES`.

**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
rotating the **shared** Gluetun rotates the exit for *all* attached
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
gateway.

> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
> containers/systemd/<unit>.container` `Environment` to inject names into the
> tunnel's `/etc/hosts`.

## Observed Dankclaw baseline (2026-08-25)

The read-only audit found:

- SearXNG image: `ghcr.io/searxng/searxng:latest`; embedded version `1.0.0`.
- Gluetun uses `VPN_SERVICE_PROVIDER=custom` with a static WireGuard profile;
  no `SERVER_COUNTRIES`, `SERVER_CITIES`, `SERVER_REGIONS`, or
  `SERVER_HOSTNAMES` filters are configured.
- The observed exit was `149.22.88.19` (Seattle, Washington, US).
- Bing and DuckDuckGo returned results. Mojeek returned no results without an
  upstream error.
- `google cse` and Brave were suspended for too many requests; Startpage was
  suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded
  Google's "unusual traffic from your network" response.
- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
  healthy. This isolates the primary failures to upstream provider handling,
  not application availability.

This deployment returns the legacy array form for `unresponsive_engines`:

```json
"unresponsive_engines": [
  ["brave", "Suspended: too many requests"],
  ["startpage", "Suspended: CAPTCHA"]
]
```

Do not assume the newer object form (`engine`, `error_type`, `suspended`)
until the live response has been checked. The direct published port is the
useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
hostname resolved to the WAN address during the audit and timed out from the
audit host, so it was not used as the baseline path.

## Fix D — Verify an engine is unblocked

Three layers, cheapest first. All queries should hit the path production uses
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).

**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
(otherwise `format=json` → 403), then query a single engine:

```bash
BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'
```

Inspect:
- `content[].engine` — which engine produced each result.
- `unresponsive_engines[]` — inspect the live shape first. On Dankclaw it is
  `[engine, message]`, for example `["google cse", "Suspended: too many requests"]`.
  Newer builds may return objects containing `engine`, `error_type`, and
  `suspended`.
- `timings[].engine` — per-engine latency when the running build emits timings.

Full status check (bash + jq):

```bash
BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
  curl -s "$BASE/search?q=test&engines=$e&format=json" \
    | jq -r --arg e "$e" '
        (.unresponsive_engines // [])
        | map(select(.engine==$e))
        | if length==0 then "\($e): OK"
          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done
```

**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.

**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance via
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.

Local self-test CLI (`kvch/searx-checker`):

```bash
python3 checker/checker.py http://192.168.0.143:8081
# google.OK  bing.OK  gitlab......ERROR
```

Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
signal), not a direct host port (which uses the host LAN IP and can be
misleading).

## Recommended path for Dankclaw (ranked)

1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
   query classes before enabling a bundled engine in default aggregation.
2. **Keep default fan-out small** — prefer independent anonymous providers and
   disable pagination for rate-sensitive engines.
3. **Promote by evidence** — raise `weight` only for engines that return useful
   results consistently; weight is not a health check.
4. **Avoid credentials** — do not add API-backed engines unless the provider is
   genuinely free, anonymous, and compatible with our privacy requirements.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
   exactly which engines are down and why. Confirm the running SearXNG version
   (check whether `parsers`/`rate_limit` exist in the pinned build).
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
   upstream request, caps it at 15 seconds, and retries once. This addresses
   modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
   after measuring the resulting error and latency profile.
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
   a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
   isolation is needed). Avoid the same-netns-proxy trap.

---

## Sources

- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.

## See Also

- [[SearXNG]] — the stack, routing, limiter, health
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
- [[Service Research Brief]] — the research that produced these pages