docs: add SearXNG providers guide for unblocking Google behind Gluetun VPN

SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP, so
rate-sensitive engines (Google, DuckDuckGo) answer with reCAPTCHA/429 and
SearXNG suspends the offending engine for hours-to-days, showing up as
"unavailable". Add docs/Dankclaw/SearXNG Providers.md covering: the
suspension model, the two failure modes (CAPTCHA vs selector drift), engine
config (settings.yml engines: list — duplicate google, timeout,
retry_on_http_error, JSON feed), request tuning (outgoing block, UA rotation,
suspended_times — correcting the old rate_limit/useragents blog myths),
routing/proxies (per-engine proxies, dedicated 2nd Gluetun, dedicated IP), and
verification (/json, /config, check.searx.space). Crosslink into SearXNG.md
and index.md.

docs/Dankclaw/SearXNG Providers.md

@@ -0,0 +1,330 @@
+# SearXNG Providers
+
+How Google and other upstream search engines get blocked behind the Gluetun
+VPN tunnel, and how to unblock them in SearXNG.
+
+**Host:** Dankclaw — **Docs:** https://docs.searxng.org
+
+SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
+Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
+Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
+start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
+engine for an hour to 15 days and silently drops it from results. That is the
+"Google (and sometimes a few others) went unavailable" symptom.
+
+This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
+are the detail if you need it.
+
+## Root cause (one line)
+
+All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
+Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
+rate-limit it → CAPTCHA → engine **suspension** → empty results.
+
+## The suspension model (what "unavailable" actually is)
+
+An engine that returns 429 / reCAPTCHA / access-denied raises a named
+exception and is *suspended* (removed from result aggregation for a set time).
+Suspended engines are the usual cause of "gone":
+
+| Exception | Default suspension | Trigger |
+|-----------|--------------------|---------|
+| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
+| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
+| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
+| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
+| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
+| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
+
+The values are `search.suspended_times` (+ `ban_time_on_fail`) in
+`settings.yml`. An engine that returns **0 results without an error** is *not*
+suspended — that is the separate "selector drift" problem (below).
+
+## Two failure modes
+
+| Mode | Symptom | Root cause | Fix family |
+|------|---------|-----------|-----------|
+| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
+| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |
+
+Most "Google is broken on a VPN host" cases are a **mix of both**.
+
+## Fix A — Engine config (settings.yml `engines:`)
+
+Important: SearXNG's engines are configured in the **`engines:` list inside
+`settings.yml`**, not a separate `engines.yml` file. With
+`use_default_settings: true` (our setting) your `engines:` list is *merged* over
+the built-in engines by `name` — you do **not** set `use_default_settings: false`.
+
+To add, override, disable, or prune:
+
+```yaml
+# shared/searxng/settings.yml  (appended to the existing file)
+use_default_settings: true
+engines:
+  # prune a broken engine entirely
+  remove:
+    - # (example) wikipedia   # drop an engine that never recovers
+
+  # override + duplicate the Google engines so one survives the other's suspension
+  - name: google
+    weight: 3
+    disabled: false
+    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
+    retry_on_http_error: [403, 429]
+    display_error_messages: true
+
+  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
+    engine: google
+    categories: general
+    weight: 2
+    timeout: 5.0
+    disabled: false
+```
+
+Key per-engine keys (each overwrites the engine-module default; see
+`settings_engines.html`):
+
+| Key | Effect |
+|-----|--------|
+| `name` | merge key (lowercase, no underscore) |
+| `engine` | Python module (`google`, `json_engine`, `xpath`) |
+| `categories` | categories it serves (also a comma string) |
+| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
+| `weight` | ranks results higher |
+| `disabled` | off by default, selectable in /preferences |
+| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
+| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
+| `display_error_messages` | show/hide the engine's error banner |
+| `proxies` | per-engine egress override (see Fix C) |
+| `language` | force region, e.g. `en` |
+
+**Best duplicate trick:** re-declare `google` under a new `name` in a
+*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
+`google2` still returns results, and vice-versa.
+
+**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
+`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
+XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
+[scrapy_html]` under `name: google` mostly does nothing. To make Google
+robust against selector drift, either update the upstream `google.py`
+selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
+Verify `parsers` consumption on the pinned version before depending on it.
+
+**JSON-feed fallback:** decouple parsing from Google's HTML churn.
+
+```yaml
+engines:
+  - name: google-json
+    engine: json_engine
+    categories: general
+    paging: true
+    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
+    results_query: results
+    title_query: title
+    url_query: link
+    content_query: snippet
+```
+
+## Fix B — Request tuning (settings.yml `outgoing:`)
+
+SearXNG already rotates a **random browser User-Agent per outgoing request**
+(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
+key**. The blogs that point at a `useragents:` table describe the old
+`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.
+
+Tune the `outgoing` block and the suspend model to survive load:
+
+```yaml
+outgoing:
+  request_timeout: 6.0             # raise from 2.0 so slow engines aren't dropped as unresponsive
+  max_request_timeout: 15.0
+  pool_connections: 200            # bigger pool = more concurrent connections per engine host
+  pool_maxsize: 40                 # keep-alive connections per host
+  enable_http2: true               # multiplexing = fewer distinct "new connection" events per engine
+  retries: 3                       # on 429/5xx, retry against a different proxy + source IP
+  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you
+
+search:
+  ban_time_on_fail: 10
+  max_ban_time_on_fail: 180
+  suspended_times:
+    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
+    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries
+```
+
+Notes:
+- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
+  `time_between_requests` sleep field does **not** exist in current SearXNG —
+  that is a historical-searx concept. The functional equivalent is the suspend
+  schedule above (widen to back off before blocking, narrow to recover faster).
+- `request_timeout` interacts with the suspend model: too tight → engines look
+  "unresponsive"; too loose → slow pages. Aim per-engine `timeout` at
+  *proxy-add-on time + engine latency*, capped by `max_request_timeout`.
+- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
+  engine return zero rows and *look* blocked.
+
+## Fix C — Routing & proxies (change which IP the upstreams exit as)
+
+Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
+strategy. The decisive fact for our topology: a proxy that runs **inside the
+same Gluetun netns** exits from the *same* single IP, so round-robin among
+same-IP proxies **does not** unblock. The proxy must egress from a **different,
+clean IP**.
+
+```mermaid
+graph LR
+    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
+    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
+```
+
+**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
+heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
+clean IP:
+
+```yaml
+engines:
+  - name: google
+    proxies:
+      http:
+        - http://gluetun-upstream:8080
+      https:
+        - http://gluetun-upstream:8080
+        - socks5h://gluetun-upstream:8388
+  - name: google news
+    engine: google_news
+    proxies:
+      https:
+        - http://gluetun-upstream:8080
+```
+
+`using_tor_proxy` follows the same per-engine/global pattern.
+
+**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
+Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
+(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
+(still on the main host/Caddy) stays put; only *upstream* requests re-route.
+Gives a second isolated exit IP and reputation pool.
+- **Cons:** a second ProtenVPN device slot, a new container, port publishing
+  moves to Gluetun, and **split-brain DNS** (below).
+
+**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
+subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
+sources reviewed (the long-requested *personal*-plan dedicated IP is still only
+on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
+via `SERVER_HOSTNAMES`.
+
+**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
+`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
+rotating the **shared** Gluetun rotates the exit for *all* attached
+containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
+gateway.
+
+> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
+> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
+> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
+> containers/systemd/<unit>.container` `Environment` to inject names into the
+> tunnel's `/etc/hosts`.
+
+## Fix D — Verify an engine is unblocked
+
+Three layers, cheapest first. All queries should hit the path production uses
+(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
+
+**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
+(otherwise `format=json` → 403), then query a single engine:
+
+```bash
+BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
+curl -s "$BASE/search?q=test&engines=google&format=json" \
+  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'
+```
+
+Inspect:
+- `content[].engine` — which engine produced each result.
+- `unresponsive_engines[]` — `{engine, error_type, suspended}`. `error_type`
+  tells you *why* (`SearxEngineTooManyRequests`, `SearxEngineCaptcha`,
+  `timeout`, `SearxEngineAccessDenied`); `suspended: true` = auto-banned right
+  now.
+- `timings[].engine` — per-engine latency.
+
+Full status check (bash + jq):
+
+```bash
+BASE="http://192.168.0.143:8081"
+for e in google bing duckduckgo brave startpage; do
+  curl -s "$BASE/search?q=test&engines=$e&format=json" \
+    | jq -r --arg e "$e" '
+        (.unresponsive_engines // [])
+        | map(select(.engine==$e))
+        | if length==0 then "\($e): OK"
+          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
+done
+```
+
+**2. `/config` REST endpoint** lists every engine + enabled/disabled +
+categories — an inventory of what is live. `?engines=<name>` isolates a single
+provider; `/debug` adds engine diagnostics.
+
+**3. `check.searx.space`** — the hosted official checker probes every engine of
+your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
+plain HTTP probe misses) and posts results on searx.space. Add your instance via
+a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
+to reach the SearXNG port.
+
+Local self-test CLI (`kvch/searx-checker`):
+
+```bash
+python3 checker/checker.py http://192.168.0.143:8081
+# google.OK  bing.OK  gitlab......ERROR
+```
+
+Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
+signal), not a direct host port (which uses the host LAN IP and can be
+misleading).
+
+## Recommended path for Dankclaw (ranked)
+
+1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
+   exactly which engines are down and why. Confirm the running SearXNG version
+   (check whether `parsers`/`rate_limit` exist in the pinned build).
+2. **Tighten Fix B** — raise `outgoing.request_timeout`, add `retry_on_http_error:
+   [403, 429]` on the Google engines, widen `suspended_times`. Cheapest, no new
+   infra.
+3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
+   other returning results.
+4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
+   dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
+   isolation is needed). Avoid the same-netns-proxy trap.
+
+---
+
+## Sources
+
+- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
+- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
+- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
+- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
+- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
+- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
+- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
+- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
+- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
+- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
+- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
+- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
+- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
+- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
+- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
+- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
+- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
+- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
+- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
+- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
+
+## See Also
+
+- [[SearXNG]] — the stack, routing, limiter, health
+- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
+- [[Dankclaw Stack]] — the service the SearXNG stack sits in
+- [[Service Research Brief]] — the research that produced these pages

docs/Dankclaw/SearXNG.md

@@ -94,7 +94,22 @@ The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The const
 
 Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
 
-## Gotchas
+## Unblocking Google & Search Providers
+
+SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The
+rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
+and SearXNG **suspends** the offending engine for hours to days, which is what
+shows up as "Google went unavailable". This is upstream, not local.
+
+Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
+`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
+fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
+if Google still re-blocks — route the Google engines at a clean-IP proxy via
+per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
+root cause, the suspend-model table, the `engines:` config, and the
+`/json` verification recipe.
+
+### Gotchas
 
 - The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
 - The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
@@ -106,4 +121,5 @@ Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local s
 
 - [[Dankclaw Stack]]
 - [[Gluetun]] — the VPN container
-- Knowledge base: [[Gluetun]], [[SearXNG]], [[Hermes]]
+- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
+- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]

docs/index.md

@@ -51,6 +51,7 @@ covers both workloads).
 ## Dankclaw Services
 
 - [[SearXNG]] — privacy metasearch (VPN-routed)
+- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
 - [[Gluetun]] — the VPN tunnel container
 - [[Valkey]] — SearXNG cache and limiter store
 - [[ComfyUI]] — node-based AI image generation (CPU)