docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

All in-repo path references (AGENTS.md, README, old scripts, report artifacts, test code paths) rewritten in the same commit; focused-test map extended for .gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's own convention ('add one before committing'). Obsidian wikilinks are title-based and unaffected.
2026-08-28 02:20:06 -07:00
1 parent 6d4a96a423
commit 5829a0bfe8
@@ -0,0 +1,400 @@
+# SearXNG Providers
+
+How Google and other upstream search engines get blocked behind the Gluetun
+VPN tunnel, and how to unblock them in SearXNG.
+
+**Host:** Dankclaw — **Docs:** https://docs.searxng.org
+
+SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
+Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
+Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
+start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
+engine for an hour to 15 days and silently drops it from results. That is the
+"Google (and sometimes a few others) went unavailable" symptom.
+
+This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
+are the detail if you need it.
+
+## Provider policy
+
+The default aggregation should maximize **independent, anonymous coverage**
+without turning one user query into a request storm. Disabled engines remain
+installed and selectable on demand; `inactive` is reserved for engines that
+should disappear completely.
+
+| Category | Default baseline | On-demand / deferred |
+|----------|------------------|----------------------|
+| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
+| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
+| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
+| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
+| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
+| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
+
+This policy uses no API credentials and does not identify the operator to a
+search provider. It is deliberately conservative: a bundled engine is not
+necessarily healthy, anonymous, or suitable for every default query.
+
+## Root cause (one line)
+
+All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
+Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
+rate-limit it → CAPTCHA → engine **suspension** → empty results.
+
+## The suspension model (what "unavailable" actually is)
+
+An engine that returns 429 / reCAPTCHA / access-denied raises a named
+exception and is *suspended* (removed from result aggregation for a set time).
+Suspended engines are the usual cause of "gone":
+
+| Exception | Default suspension | Trigger |
+|-----------|--------------------|---------|
+| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
+| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
+| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
+| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
+| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
+| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
+
+The values are `search.suspended_times` (+ `ban_time_on_fail`) in
+`settings.yml`. An engine that returns **0 results without an error** is *not*
+suspended — that is the separate "selector drift" problem (below).
+
+## Two failure modes
+
+| Mode | Symptom | Root cause | Fix family |
+|------|---------|-----------|-----------|
+| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
+| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |
+
+Most "Google is broken on a VPN host" cases are a **mix of both**.
+
+## Fix A — Engine config (settings.yml `engines:`)
+
+The current tracked policy enables Bing and DuckDuckGo for default general
+search, gives each weight 2, disables pagination, and allows 5 seconds for the
+VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
+but disabled from default aggregation. This limits fan-out while preserving
+explicit/on-demand selection.
+
+Important: SearXNG's engines are configured in the **`engines:` list inside
+`settings.yml`**, not a separate `engines.yml` file. With
+`use_default_settings: true` (our setting) your `engines:` list is *merged* over
+the built-in engines by `name` — you do **not** set `use_default_settings: false`.
+
+To add, override, disable, or prune:
+
+```yaml
+# shared/searxng/settings.yml  (appended to the existing file)
+use_default_settings: true
+engines:
+  # prune a broken engine entirely
+  remove:
+    - # (example) wikipedia   # drop an engine that never recovers
+
+  # override + duplicate the Google engines so one survives the other's suspension
+  - name: google
+    weight: 3
+    disabled: false
+    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
+    retry_on_http_error: [403, 429]
+    display_error_messages: true
+
+  - name: google2                # fresh engine, same module, survives google's CAPTCHA ban
+    engine: google
+    categories: general
+    weight: 2
+    timeout: 5.0
+    disabled: false
+```
+
+Key per-engine keys (each overwrites the engine-module default; see
+`settings_engines.html`):
+
+| Key | Effect |
+|-----|--------|
+| `name` | merge key (lowercase, no underscore) |
+| `engine` | Python module (`google`, `json_engine`, `xpath`) |
+| `categories` | categories it serves (also a comma string) |
+| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
+| `weight` | ranks results higher |
+| `disabled` | off by default, selectable in /preferences |
+| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
+| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
+| `display_error_messages` | show/hide the engine's error banner |
+| `proxies` | per-engine egress override (see Fix C) |
+| `language` | force region, e.g. `en` |
+
+**Best duplicate trick:** re-declare `google` under a new `name` in a
+*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
+`google2` still returns results, and vice-versa.
+
+**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
+`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
+XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
+[scrapy_html]` under `name: google` mostly does nothing. To make Google
+robust against selector drift, either update the upstream `google.py`
+selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
+Verify `parsers` consumption on the pinned version before depending on it.
+
+**JSON-feed fallback:** decouple parsing from Google's HTML churn.
+
+```yaml
+engines:
+  - name: google-json
+    engine: json_engine
+    categories: general
+    paging: true
+    search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
+    results_query: results
+    title_query: title
+    url_query: link
+    content_query: snippet
+```
+
+## Fix B — Request tuning (settings.yml `outgoing:`)
+
+SearXNG already rotates a **random browser User-Agent per outgoing request**
+(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
+key**. The blogs that point at a `useragents:` table describe the old
+`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.
+
+Tune the `outgoing` block and the suspend model to survive load:
+
+```yaml
+outgoing:
+  request_timeout: 4.0             # current Dankclaw value; modest VPN-latency allowance
+  max_request_timeout: 12.0
+  pool_connections: 100
+  pool_maxsize: 20
+  enable_http2: true
+  retries: 1                       # avoid amplifying 429/CAPTCHA responses from one exit IP
+  useragent_suffix: "ops@yourhomelab.example"   # so a per-engine blocker can reach you
+
+search:
+  ban_time_on_fail: 10
+  max_ban_time_on_fail: 180
+  suspended_times:
+    SearxEngineTooManyRequests: 300     # 429 → 5 min before it retries (tune down to recover faster)
+    SearxEngineCaptcha: 7200            # CAPTCHA → 2h before it retries
+```
+
+Notes:
+- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
+  `time_between_requests` sleep field does **not** exist in current SearXNG —
+  that is a historical-searx concept. The functional equivalent is the suspend
+  schedule above (widen to back off before blocking, narrow to recover faster).
+- `request_timeout` interacts with the suspend model: too tight → engines look
+  "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
+  12 seconds as the cap; raise it only when measurements show real upstream
+  latency rather than 429/CAPTCHA responses.
+- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
+  provider's rate limit worse. A retry count of 1 is a bounded compromise for a
+  transient network failure.
+- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
+  engine return zero rows and *look* blocked.
+
+## Fix C — Routing & proxies (change which IP the upstreams exit as)
+
+Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
+strategy. The decisive fact for our topology: a proxy that runs **inside the
+same Gluetun netns** exits from the *same* single IP, so round-robin among
+same-IP proxies **does not** unblock. The proxy must egress from a **different,
+clean IP**.
+
+```mermaid
+graph LR
+    SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
+    SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
+```
+
+**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
+heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
+clean IP:
+
+```yaml
+engines:
+  - name: google
+    proxies:
+      http:
+        - http://gluetun-upstream:8080
+      https:
+        - http://gluetun-upstream:8080
+        - socks5h://gluetun-upstream:8388
+  - name: google news
+    engine: google_news
+    proxies:
+      https:
+        - http://gluetun-upstream:8080
+```
+
+`using_tor_proxy` follows the same per-engine/global pattern.
+
+**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
+Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
+(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
+(still on the main host/Caddy) stays put; only *upstream* requests re-route.
+Gives a second isolated exit IP and reputation pool.
+- **Cons:** a second ProtenVPN device slot, a new container, port publishing
+  moves to Gluetun, and **split-brain DNS** (below).
+
+**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
+subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
+sources reviewed (the long-requested *personal*-plan dedicated IP is still only
+on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
+via `SERVER_HOSTNAMES`.
+
+**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
+`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
+rotating the **shared** Gluetun rotates the exit for *all* attached
+containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
+gateway.
+
+> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
+> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
+> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
+> containers/systemd/<unit>.container` `Environment` to inject names into the
+> tunnel's `/etc/hosts`.
+
+## Observed Dankclaw baseline (2026-08-25)
+
+The read-only audit found:
+
+- SearXNG image: `ghcr.io/searxng/searxng:latest`; embedded version `1.0.0`.
+- Gluetun uses `VPN_SERVICE_PROVIDER=custom` with a static WireGuard profile;
+  no `SERVER_COUNTRIES`, `SERVER_CITIES`, `SERVER_REGIONS`, or
+  `SERVER_HOSTNAMES` filters are configured.
+- The observed exit was `149.22.88.19` (Seattle, Washington, US).
+- Bing and DuckDuckGo returned results. Mojeek returned no results without an
+  upstream error.
+- `google cse` and Brave were suspended for too many requests; Startpage was
+  suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded
+  Google's "unusual traffic from your network" response.
+- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
+  healthy. This isolates the primary failures to upstream provider handling,
+  not application availability.
+
+This deployment returns the legacy array form for `unresponsive_engines`:
+
+```json
+"unresponsive_engines": [
+  ["brave", "Suspended: too many requests"],
+  ["startpage", "Suspended: CAPTCHA"]
+]
+```
+
+Do not assume the newer object form (`engine`, `error_type`, `suspended`)
+until the live response has been checked. The direct published port is the
+useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
+hostname resolved to the WAN address during the audit and timed out from the
+audit host, so it was not used as the baseline path.
+
+## Fix D — Verify an engine is unblocked
+
+Three layers, cheapest first. All queries should hit the path production uses
+(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
+
+**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
+(otherwise `format=json` → 403), then query a single engine:
+
+```bash
+BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
+curl -s "$BASE/search?q=test&engines=google&format=json" \
+  | jq -r '.unresponsive_engines[]? | select(.engine=="google")'
+```
+
+Inspect:
+- `content[].engine` — which engine produced each result.
+- `unresponsive_engines[]` — inspect the live shape first. On Dankclaw it is
+  `[engine, message]`, for example `["google cse", "Suspended: too many requests"]`.
+  Newer builds may return objects containing `engine`, `error_type`, and
+  `suspended`.
+- `timings[].engine` — per-engine latency when the running build emits timings.
+
+Full status check (bash + jq):
+
+```bash
+BASE="http://192.168.0.143:8081"
+for e in google bing duckduckgo brave startpage; do
+  curl -s "$BASE/search?q=test&engines=$e&format=json" \
+    | jq -r --arg e "$e" '
+        (.unresponsive_engines // [])
+        | map(select(.engine==$e))
+        | if length==0 then "\($e): OK"
+          else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
+done
+```
+
+**2. `/config` REST endpoint** lists every engine + enabled/disabled +
+categories — an inventory of what is live. `?engines=<name>` isolates a single
+provider; `/debug` adds engine diagnostics.
+
+**3. `check.searx.space`** — the hosted official checker probes every engine of
+your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
+plain HTTP probe misses) and posts results on searx.space. Add your instance via
+a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
+to reach the SearXNG port.
+
+Local self-test CLI (`kvch/searx-checker`):
+
+```bash
+python3 checker/checker.py http://192.168.0.143:8081
+# google.OK  bing.OK  gitlab......ERROR
+```
+
+Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
+signal), not a direct host port (which uses the host LAN IP and can be
+misleading).
+
+## Recommended path for Dankclaw (ranked)
+
+1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
+   query classes before enabling a bundled engine in default aggregation.
+2. **Keep default fan-out small** — prefer independent anonymous providers and
+   disable pagination for rate-sensitive engines.
+3. **Promote by evidence** — raise `weight` only for engines that return useful
+   results consistently; weight is not a health check.
+4. **Avoid credentials** — do not add API-backed engines unless the provider is
+   genuinely free, anonymous, and compatible with our privacy requirements.
+5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
+   exactly which engines are down and why. Confirm the running SearXNG version
+   (check whether `parsers`/`rate_limit` exist in the pinned build).
+6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
+   upstream request, caps it at 15 seconds, and retries once. This addresses
+   modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
+   after measuring the resulting error and latency profile.
+7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
+   a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
+   isolation is needed). Avoid the same-netns-proxy trap.
+
+---
+
+## Sources
+
+- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
+- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
+- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
+- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
+- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
+- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
+- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
+- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
+- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
+- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
+- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
+- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
+- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
+- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
+- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
+- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
+- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
+- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
+- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
+- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
+
+## See Also
+
+- [[SearXNG]] — the stack, routing, limiter, health
+- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
+- [[Dankclaw Stack]] — the service the SearXNG stack sits in
+- [[Service Research Brief]] — the research that produced these pages