serverAppConfigs/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md

---
type: reference
description: "How Google and other upstream search engines get blocked behind the Gluetun"
tags:
  - dankclaw
  - searxng
timestamp: 2026-08-28
---

# SearXNG Providers

How Google and other upstream search engines get blocked behind the Gluetun
VPN tunnel, and how to unblock them in SearXNG.

**Host:** Dankclaw — **Docs:** https://docs.searxng.org

SearXNG aggregates the engines bundled in the running build — **262 installed, 75
enabled** on the live instance (`GET http://127.0.0.1:8081/config`, 2026-10-18; the
"~50 engines" framing this page carried predates that count). When all of them are
queried through **one** VPN exit IP — as on
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
engine (three minutes for a 429 under the current build's defaults, up to 15 days for
a Cloudflare CAPTCHA — see the table below) and silently drops it from results. That
is the "Google (and sometimes a few others) went unavailable" symptom.

This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.

## Provider policy

The default aggregation should maximize **independent, anonymous coverage**
without turning one user query into a request storm. Disabled engines remain
installed and selectable on demand; `inactive` is reserved for engines that
should disappear completely.

| Category | Default baseline | On-demand / deferred |
|----------|------------------|----------------------|
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
| Science | arXiv, PubMed, Semantic Scholar | Google Scholar until independently tested; OpenAlex (`openalex` is `enabled: false` in the running build, `/config` 2026-10-18) |
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
| Images | Wikimedia Commons (ships as `wikicommons.images/videos/audio/files`, all enabled), Bing Images, Unsplash | Openverse (`enabled: false` today), Google/Brave/Startpage image engines |
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |

**Live state of this table (2026-10-18, from `GET /config`):** the General/web,
Technical, News and Videos baselines hold engine-for-engine; the two rows above are
the corrections. Nothing in the deferred column is enabled. One operational note for
the deferred engines: re-enabling Google CSE on demand needs a key this host does not
hold — `/opt/searxng/.env` carries only the WireGuard variables plus
`SEARXNG_SECRET`, `SEARXNG_BASE_URL` and `KAGI_API_KEY` (key **names** read
2026-10-18; values not printed).

This policy was written to use no API credentials and to avoid identifying the
operator to a search provider — **with one carve-out made afterwards (2026-09-26):
the paid `kagi` engine, whose key is rendered into the deployed config from
`/opt/searxng/.env` (`shared/searxng/settings.yml:70`; item 4 below and
[[SearXNG]]).** It is deliberately conservative: a bundled engine is not
necessarily healthy, anonymous, or suitable for every default query.

## Root cause (one line)

All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
Google/DuckDuckGo see one datacenter IP spamming every enabled engine (75 of the 262
installed ones, `/config` 2026-10-18) and rate-limit it → CAPTCHA → engine
**suspension** → empty results.

## The suspension model (what "unavailable" actually is)

An engine that returns 429 / reCAPTCHA / access-denied raises a named
exception and is *suspended* (removed from result aggregation for a set time).
Suspended engines are the usual cause of "gone":

| Exception | Default suspension | Trigger |
|-----------|--------------------|---------|
| `SearxEngineTooManyRequests` | **180 s (3 min)** | HTTP 429 |
| `SearxEngineAccessDenied` | **180 s (3 min)** | 403 / access denied |
| `SearxEngineCaptcha` | **3600 s (1 h)** | Google `/sorry/` or reCAPTCHA |
| `recaptcha_SearxEngineCaptcha` | ~7 days (604 800 s) | Google reCAPTCHA |
| `cf_SearxEngineCaptcha` | ~15 days (1 296 000 s) | Cloudflare edge CAPTCHA |
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail` 5 / `max_ban_time_on_fail` 120 |

These are the defaults of the **running commit** `12f8b6515`, read from
[`searx/settings.yml` at that commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml)
(lines 66-81). The folklore numbers this table used to carry — ~1 hour for 429, ~1 day
for access-denied and CAPTCHA — were the **older** upstream defaults and are retired;
the current build un-suspends a 429'd engine after three minutes.

The values are `search.suspended_times` (+ `ban_time_on_fail`). Our
`shared/searxng/settings.yml` overrides **none** of them (`grep -n "suspended_times\|ban_time_on_fail"`
returns nothing, 2026-10-18), so the upstream defaults quoted above are exactly what
runs. An engine that returns **0 results without an error** is *not*
suspended — that is the separate "selector drift" problem (below).

## Two failure modes

| Mode | Symptom | Root cause | Fix family |
|------|---------|-----------|-----------|
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |

Most "Google is broken on a VPN host" cases are a **mix of both**.

## Fix A — Engine config (settings.yml `engines:`)

The current tracked policy enables Bing and DuckDuckGo for default general
search, gives each weight 2, disables pagination, and allows 5.0 seconds for the
VPN path (`shared/searxng/settings.yml:26,38-56`). CAPTCHA/429-prone Google, Brave,
and Startpage families are retained but disabled from default aggregation
(`:81-124`) — though the three `startpage*` stanzas are dead weight: the running
build ships no `startpage` engine at all (`/config` lists none, 2026-10-18), so those
overrides match nothing. This limits fan-out while preserving
explicit/on-demand selection.

Important: SearXNG's engines are configured in the **`engines:` list inside
`settings.yml`**, not a separate `engines.yml` file. With
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
the built-in engines by `name` — you do **not** set `use_default_settings: false`.

To add, override, disable, or prune:

```yaml
# shared/searxng/settings.yml  (appended to the existing file)
use_default_settings: true
engines:
  # prune a broken engine entirely
  remove:
    - # (example) wikipedia   # drop an engine that never recovers

  # override + duplicate the Google engines so one survives the other's suspension
  - name: google
    weight: 3
    disabled: false
    timeout: 5.0                 # wait longer over the tunnel (upstream global default is 3.0s — searx/settings.yml:178 at the running commit)
    retry_on_http_error: [403, 429]
    display_error_messages: true

  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
    engine: google
    categories: general
    weight: 2
    timeout: 5.0
    disabled: false
```

Key per-engine keys (each overwrites the engine-module default
`settings_engines.html`):

| Key | Effect |
|-----|--------|
| `name` | merge key (lowercase, no underscore) |
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
| `categories` | categories it serves (also a comma string) |
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
| `weight` | ranks results higher |
| `disabled` | off by default, selectable in /preferences |
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
| `display_error_messages` | show/hide the engine's error banner |
| `proxies` | per-engine egress override (see Fix C) |
| `language` | force region, e.g. `en` |

**Best duplicate trick:** re-declare `google` under a new `name` in a
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
`google2` still returns results, and vice-versa.

**Parser caveat (read before fiddling with `parsers`):** the modern `google`
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
[scrapy_html]` under `name: google` mostly does nothing. To make Google
robust against selector drift, either update the upstream `google.py`
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
Verify `parsers` consumption on the pinned version before depending on it.

**JSON-feed fallback:** decouple parsing from Google's HTML churn.

```yaml
engines:
  - name: google-json
    engine: json_engine
    categories: general
    paging: true
    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
    results_query: results
    title_query: title
    url_query: link
    content_query: snippet
```

## Fix B — Request tuning (settings.yml `outgoing:`)

SearXNG already rotates a **random browser User-Agent per outgoing request**
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
key**. The blogs that point at a `useragents:` table describe the old
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.

Tune the `outgoing` block and the suspend model to survive load:

```yaml
outgoing:
  request_timeout: 5.0             # Dankclaw's value (shared/searxng/settings.yml:26)
  max_request_timeout: 15.0        # Dankclaw's cap (shared/searxng/settings.yml:27)
  pool_connections: 100
  pool_maxsize: 20
  enable_http2: true
  retries: 1                       # avoid amplifying 429/CAPTCHA responses from one exit IP
  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you

search:
  ban_time_on_fail: 10
  max_ban_time_on_fail: 180
  suspended_times:
    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries
```

Notes:
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
  `time_between_requests` sleep field does **not** exist in current SearXNG —
  that is a historical-searx concept. The functional equivalent is the suspend
  schedule above (widen to back off before blocking, narrow to recover faster).
- `request_timeout` interacts with the suspend model: too tight → engines look
  "unresponsive"; too loose → slow pages. Dankclaw uses 5 seconds globally and
  15 seconds as the cap (`shared/searxng/settings.yml:26-27`); raise it only when
  measurements show real upstream latency rather than 429/CAPTCHA responses.
- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
  provider's rate limit worse. A retry count of 1 is a bounded compromise for a
  transient network failure.
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
  engine return zero rows and *look* blocked.

## Fix C — Routing & proxies (change which IP the upstreams exit as)

Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
strategy. The decisive fact for our topology: a proxy that runs **inside the
same Gluetun netns** exits from the *same* single IP, so round-robin among
same-IP proxies **does not** unblock. The proxy must egress from a **different,
clean IP**.

```mermaid
graph LR
    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
```

**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
clean IP:

```yaml
engines:
  - name: google
    proxies:
      http:
        - http://gluetun-upstream:8080
      https:
        - http://gluetun-upstream:8080
        - socks5h://gluetun-upstream:8388
  - name: google news
    engine: google_news
    proxies:
      https:
        - http://gluetun-upstream:8080
```

`using_tor_proxy` follows the same per-engine/global pattern.

**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
Gives a second isolated exit IP and reputation pool.
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
  moves to Gluetun, and **split-brain DNS** (below).

**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
via `SERVER_HOSTNAMES`.

**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
rotating the **shared** Gluetun rotates the exit for *all* attached
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
gateway.

> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
> containers/systemd/<unit>.container` `Environment` to inject names into the
> tunnel's `/etc/hosts`.

## Observed Dankclaw baseline (2026-08-25)

The read-only audit found:

- SearXNG image: `ghcr.io/searxng/searxng:latest`; embedded version `1.0.0`.
- Gluetun uses `VPN_SERVICE_PROVIDER=custom` with a static WireGuard profile;
  no `SERVER_COUNTRIES`, `SERVER_CITIES`, `SERVER_REGIONS`, or
  `SERVER_HOSTNAMES` filters are configured.
- The observed exit was `149.22.88.19` (Seattle, Washington, US).
- Bing and DuckDuckGo returned results. Mojeek returned no results without an
  upstream error.
- `google cse` and Brave were suspended for too many requests; Startpage was
  suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded
  Google's "unusual traffic from your network" response.
- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
  healthy. This isolates the primary failures to upstream provider handling,
  not application availability. Re-checked 2026-10-18: `/healthz` is still `200`
  and `gluetun`/`searxng`/`searxng-valkey` are all `generated` + `active`.

What has moved since that audit: the container is still
`ghcr.io/searxng/searxng:latest` (`podman inspect searxng` 2026-10-18), but the
embedded version is now `2026.9.25+12f8b6515` (`GET /config`), not `1.0.0`, and
`limiter.enabled` is `true`. The exit IP was not re-measured (that would need a
query fired through the tunnel).

This deployment returns the legacy array form for `unresponsive_engines`:

```json
"unresponsive_engines": [
  ["brave", "Suspended: too many requests"],
  ["startpage", "Suspended: CAPTCHA"]
]
```

Do not assume the newer object form (`engine`, `error_type`, `suspended`)
until the live response has been checked — and note that the jq recipes below key on
`.engine`, which only matches the object form; against the legacy `[engine, message]`
pair they silently match nothing. *(Re-checked 2026-10-18 with a `bing`-only JSON
query: `unresponsive_engines` came back empty, so today's shape is unverified.)* The
direct published port is the useful diagnostic path because it exercises the
ProtonVPN exit. The Caddy hostname resolved to the WAN address during the audit and
timed out from the audit host, so it was not used as the baseline path; on
2026-10-18 `search.franksdank.cloud` resolves to the LAN IP from inside the network
and answers with the Anubis challenge page.

## Fix D — Verify an engine is unblocked

Three layers, cheapest first. All queries should hit the path production uses
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).

**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
(otherwise `format=json` → 403) — `html` and `json` are both enabled here
(`shared/searxng/settings.yml:18-20`), and a JSON query through the published port
returns 200 `application/json` today (2026-10-18). Query a single engine:

```bash
BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
curl -s "$BASE/search?q=test&engines=google&format=json" \
  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'
```

Inspect:
- `content[].engine` — which engine produced each result.
- `unresponsive_engines[]` — inspect the live shape first. On Dankclaw it is
  `[engine, message]`, for example `["google cse", "Suspended: too many requests"]`.
  Newer builds may return objects containing `engine`, `error_type`, and
  `suspended`.
- `timings[].engine` — per-engine latency when the running build emits timings.

Full status check (bash + jq):

```bash
BASE="http://192.168.0.143:8081"
for e in google bing duckduckgo brave startpage; do
  curl -s "$BASE/search?q=test&engines=$e&format=json" \
    | jq -r --arg e "$e" '
        (.unresponsive_engines // [])
        | map(select(.engine==$e))
        | if length==0 then "\($e): OK"
          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
done
```

**2. `/config` REST endpoint** lists every engine + enabled/disabled +
categories — an inventory of what is live. It answers 200 on this instance today
(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`
`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
[[SearXNG]]). `?engines=<name>` isolates a single
provider; `/debug` adds engine diagnostics.

**3. `check.searx.space`** — the hosted official checker probes every engine of
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
plain HTTP probe misses) and posts results on searx.space. Add your instance
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
to reach the SearXNG port.

Local self-test CLI (`kvch/searx-checker`):

```bash
python3 checker/checker.py http://192.168.0.143:8081
# google.OK  bing.OK  gitlab......ERROR
```

Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
signal), not a direct host port (which uses the host LAN IP and can be
misleading).

## Recommended path for Dankclaw (ranked)

1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
   query classes before enabling a bundled engine in default aggregation.
2. **Keep default fan-out small** — prefer independent anonymous providers and
   disable pagination for rate-sensitive engines.
3. **Promote by evidence** — raise `weight` only for engines that return useful
   results consistently; weight is not a health check.
4. **Avoid credentials, with one carve-out** — do not add API-backed engines
   unless the provider is genuinely free, anonymous, and compatible with our
   privacy requirements. Carve-out (2026-09-26): a paid, user-ordered API
   engine (Kagi) is acceptable when the blast radius is bounded — the
   `kagi` engine runs with `paging: false` and weight 2, so it spends exactly
   one API credit per aggregated query, and its key never enters git
   (deploy-time rendering from `/opt/searxng/.env` into
   `/opt/searxng/config/settings.yml`, where the `${KAGI_API_KEY}` placeholder is
   already replaced — checked 2026-10-18; see [[SearXNG]]). It is
   also immune to the shared-VPN-exit-IP blocking this page is about: it
   authenticates, so reputation of the exit IP is irrelevant.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to
   exactly which engines are down and why. Confirm the running SearXNG version
   (check whether `parsers`/`rate_limit` exist in the pinned build).
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds
   upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
   (`:27`), and retries once (`:32`). This addresses
   modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
   after measuring the resulting error and latency profile.
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
   a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
   isolation is needed). Avoid the same-netns-proxy trap.

---

## Sources

- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
- [Upstream `searx/settings.yml` at the running commit `12f8b6515`](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml) — the `suspended_times`, `ban_time_on_fail` and `request_timeout` defaults quoted above.
- [Upstream `searx/engines/kagi.py` at the same commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/engines/kagi.py) — Bearer auth, `POST https://kagi.com/api/v1/search`, `require_api_key: True`.

## See Also

- [[SearXNG]] — the stack, routing, limiter, health
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
- [[Service Research Brief]] — the research that produced these pages