diff --git a/docs/Dankclaw/SearXNG Providers.md b/docs/Dankclaw/SearXNG Providers.md new file mode 100644 index 0000000..a159aa6 --- /dev/null +++ b/docs/Dankclaw/SearXNG Providers.md @@ -0,0 +1,330 @@ +# SearXNG Providers + +How Google and other upstream search engines get blocked behind the Gluetun +VPN tunnel, and how to unblock them in SearXNG. + +**Host:** Dankclaw — **Docs:** https://docs.searxng.org + +SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave, +Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on +Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines +start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending +engine for an hour to 15 days and silently drops it from results. That is the +"Google (and sometimes a few others) went unavailable" symptom. + +This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below +are the detail if you need it. + +## Root cause (one line) + +All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so +Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and +rate-limit it → CAPTCHA → engine **suspension** → empty results. + +## The suspension model (what "unavailable" actually is) + +An engine that returns 429 / reCAPTCHA / access-denied raises a named +exception and is *suspended* (removed from result aggregation for a set time). +Suspended engines are the usual cause of "gone": + +| Exception | Default suspension | Trigger | +|-----------|--------------------|---------| +| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 | +| `SearxEngineAccessDenied` | ~1 day | 403 / access denied | +| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA | +| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA | +| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA | +| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` | + +The values are `search.suspended_times` (+ `ban_time_on_fail`) in +`settings.yml`. An engine that returns **0 results without an error** is *not* +suspended — that is the separate "selector drift" problem (below). + +## Two failure modes + +| Mode | Symptom | Root cause | Fix family | +|------|---------|-----------|-----------| +| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) | +| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) | + +Most "Google is broken on a VPN host" cases are a **mix of both**. + +## Fix A — Engine config (settings.yml `engines:`) + +Important: SearXNG's engines are configured in the **`engines:` list inside +`settings.yml`**, not a separate `engines.yml` file. With +`use_default_settings: true` (our setting) your `engines:` list is *merged* over +the built-in engines by `name` — you do **not** set `use_default_settings: false`. + +To add, override, disable, or prune: + +```yaml +# shared/searxng/settings.yml (appended to the existing file) +use_default_settings: true +engines: + # prune a broken engine entirely + remove: + - # (example) wikipedia # drop an engine that never recovers + + # override + duplicate the Google engines so one survives the other's suspension + - name: google + weight: 3 + disabled: false + timeout: 5.0 # wait longer over the tunnel (global default is 2.0s) + retry_on_http_error: [403, 429] + display_error_messages: true + + - name: google2 # fresh engine, same module, survives google's CAPTCHA ban + engine: google + categories: general + weight: 2 + timeout: 5.0 + disabled: false +``` + +Key per-engine keys (each overwrites the engine-module default; see +`settings_engines.html`): + +| Key | Effect | +|-----|--------| +| `name` | merge key (lowercase, no underscore) | +| `engine` | Python module (`google`, `json_engine`, `xpath`) | +| `categories` | categories it serves (also a comma string) | +| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` | +| `weight` | ranks results higher | +| `disabled` | off by default, selectable in /preferences | +| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine | +| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` | +| `display_error_messages` | show/hide the engine's error banner | +| `proxies` | per-engine egress override (see Fix C) | +| `language` | force region, e.g. `en` | + +**Best duplicate trick:** re-declare `google` under a new `name` in a +*different* category. When the primary `google` gets CAPTCHA'd it suspends, but +`google2` still returns results, and vice-versa. + +**Parser caveat (read before fiddling with `parsers`):** the modern `google`, +`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded +XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers: +[scrapy_html]` under `name: google` mostly does nothing. To make Google +robust against selector drift, either update the upstream `google.py` +selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed. +Verify `parsers` consumption on the pinned version before depending on it. + +**JSON-feed fallback:** decouple parsing from Google's HTML churn. + +```yaml +engines: + - name: google-json + engine: json_engine + categories: general + paging: true + search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}' + results_query: results + title_query: title + url_query: link + content_query: snippet +``` + +## Fix B — Request tuning (settings.yml `outgoing:`) + +SearXNG already rotates a **random browser User-Agent per outgoing request** +(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping +key**. The blogs that point at a `useragents:` table describe the old +`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it. + +Tune the `outgoing` block and the suspend model to survive load: + +```yaml +outgoing: + request_timeout: 6.0 # raise from 2.0 so slow engines aren't dropped as unresponsive + max_request_timeout: 15.0 + pool_connections: 200 # bigger pool = more concurrent connections per engine host + pool_maxsize: 40 # keep-alive connections per host + enable_http2: true # multiplexing = fewer distinct "new connection" events per engine + retries: 3 # on 429/5xx, retry against a different proxy + source IP + useragent_suffix: "ops@yourhomelab.example" # so a per-engine blocker can reach you + +search: + ban_time_on_fail: 10 + max_ban_time_on_fail: 180 + suspended_times: + SearxEngineTooManyRequests: 300 # 429 → 5 min before it retries (tune down to recover faster) + SearxEngineCaptcha: 7200 # CAPTCHA → 2h before it retries +``` + +Notes: +- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` / + `time_between_requests` sleep field does **not** exist in current SearXNG — + that is a historical-searx concept. The functional equivalent is the suspend + schedule above (widen to back off before blocking, narrow to recover faster). +- `request_timeout` interacts with the suspend model: too tight → engines look + "unresponsive"; too loose → slow pages. Aim per-engine `timeout` at + *proxy-add-on time + engine latency*, capped by `max_request_timeout`. +- `safe_search: 0` when diagnosing — a too-strict global safe search can make an + engine return zero rows and *look* blocked. + +## Fix C — Routing & proxies (change which IP the upstreams exit as) + +Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN +strategy. The decisive fact for our topology: a proxy that runs **inside the +same Gluetun netns** exits from the *same* single IP, so round-robin among +same-IP proxies **does not** unblock. The proxy must egress from a **different, +clean IP**. + +```mermaid +graph LR + SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked] + SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean] +``` + +**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the +heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a +clean IP: + +```yaml +engines: + - name: google + proxies: + http: + - http://gluetun-upstream:8080 + https: + - http://gluetun-upstream:8080 + - socks5h://gluetun-upstream:8388 + - name: google news + engine: google_news + proxies: + https: + - http://gluetun-upstream:8080 +``` + +`using_tor_proxy` follows the same per-engine/global pattern. + +**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second +Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy +(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound +(still on the main host/Caddy) stays put; only *upstream* requests re-route. +Gives a second isolated exit IP and reputation pool. +- **Cons:** a second ProtenVPN device slot, a new container, port publishing + moves to Gluetun, and **split-brain DNS** (below). + +**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other +subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the +sources reviewed (the long-requested *personal*-plan dedicated IP is still only +on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node +via `SERVER_HOSTNAMES`. + +**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/ +`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat: +rotating the **shared** Gluetun rotates the exit for *all* attached +containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated +gateway. + +> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns +> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it +> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/ +> containers/systemd/.container` `Environment` to inject names into the +> tunnel's `/etc/hosts`. + +## Fix D — Verify an engine is unblocked + +Three layers, cheapest first. All queries should hit the path production uses +(the **published Gluetun port → ProtonVPN exit**, not a direct host port). + +**1. Isolate one engine with `/json`.** Enable `json` in `search.formats` +(otherwise `format=json` → 403), then query a single engine: + +```bash +BASE="http://192.168.0.143:8081" # published port (ProtonVPN exit) +curl -s "$BASE/search?q=test&engines=google&format=json" \ + | jq -r '.unresponsive_engines[]? | select(.engine=="google")' +``` + +Inspect: +- `content[].engine` — which engine produced each result. +- `unresponsive_engines[]` — `{engine, error_type, suspended}`. `error_type` + tells you *why* (`SearxEngineTooManyRequests`, `SearxEngineCaptcha`, + `timeout`, `SearxEngineAccessDenied`); `suspended: true` = auto-banned right + now. +- `timings[].engine` — per-engine latency. + +Full status check (bash + jq): + +```bash +BASE="http://192.168.0.143:8081" +for e in google bing duckduckgo brave startpage; do + curl -s "$BASE/search?q=test&engines=$e&format=json" \ + | jq -r --arg e "$e" ' + (.unresponsive_engines // []) + | map(select(.engine==$e)) + | if length==0 then "\($e): OK" + else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end' +done +``` + +**2. `/config` REST endpoint** lists every engine + enabled/disabled + +categories — an inventory of what is live. `?engines=` isolates a single +provider; `/debug` adds engine diagnostics. + +**3. `check.searx.space`** — the hosted official checker probes every engine of +your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a +plain HTTP probe misses) and posts results on searx.space. Add your instance via +a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs +to reach the SearXNG port. + +Local self-test CLI (`kvch/searx-checker`): + +```bash +python3 checker/checker.py http://192.168.0.143:8081 +# google.OK bing.OK gitlab......ERROR +``` + +Verify through the **published Gluetun port** (realistic ProtonVPN-exit block +signal), not a direct host port (which uses the host LAN IP and can be +misleading). + +## Recommended path for Dankclaw (ranked) + +1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see + exactly which engines are down and why. Confirm the running SearXNG version + (check whether `parsers`/`rate_limit` exist in the pinned build). +2. **Tighten Fix B** — raise `outgoing.request_timeout`, add `retry_on_http_error: + [403, 429]` on the Google engines, widen `suspended_times`. Cheapest, no new + infra. +3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the + other returning results. +4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a + dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation + isolation is needed). Avoid the same-netns-proxy trap. + +--- + +## Sources + +- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys. +- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`. +- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`. +- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics. +- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`. +- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config. +- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings. +- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable". +- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature. +- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515) +- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP. +- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`. +- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics. +- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers. +- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs. +- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice). +- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk. +- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`. +- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun. +- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix. + +## See Also + +- [[SearXNG]] — the stack, routing, limiter, health +- [[Gluetun]] — the VPN tunnel (where the exit IP comes from) +- [[Dankclaw Stack]] — the service the SearXNG stack sits in +- [[Service Research Brief]] — the research that produced these pages diff --git a/docs/Dankclaw/SearXNG.md b/docs/Dankclaw/SearXNG.md index 59e3bff..fbfaedc 100644 --- a/docs/Dankclaw/SearXNG.md +++ b/docs/Dankclaw/SearXNG.md @@ -94,7 +94,22 @@ The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The const Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots. -## Gotchas +## Unblocking Google & Search Providers + +SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The +rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429, +and SearXNG **suspends** the offending engine for hours to days, which is what +shows up as "Google went unavailable". This is upstream, not local. + +Fixes (cheapest first): tune the `outgoing` block (`request_timeout`, +`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a +fresh `name` so a CAPTCHA ban on one leaves the other returning results, and — +if Google still re-blocks — route the Google engines at a clean-IP proxy via +per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full +root cause, the suspend-model table, the `engines:` config, and the +`/json` verification recipe. + +### Gotchas - The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`. - The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host. @@ -106,4 +121,5 @@ Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local s - [[Dankclaw Stack]] - [[Gluetun]] — the VPN container -- Knowledge base: [[Gluetun]], [[SearXNG]], [[Hermes]] +- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification) +- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]] diff --git a/docs/index.md b/docs/index.md index c05b07d..1c64ad0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -51,6 +51,7 @@ covers both workloads). ## Dankclaw Services - [[SearXNG]] — privacy metasearch (VPN-routed) +- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN - [[Gluetun]] — the VPN tunnel container - [[Valkey]] — SearXNG cache and limiter store - [[ComfyUI]] — node-based AI image generation (CPU)