diff --git a/docs/Dankclaw/SearXNG Providers.md b/docs/Dankclaw/SearXNG Providers.md index 79b8f9f..1f6acb7 100644 --- a/docs/Dankclaw/SearXNG Providers.md +++ b/docs/Dankclaw/SearXNG Providers.md @@ -15,6 +15,26 @@ engine for an hour to 15 days and silently drops it from results. That is the This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below are the detail if you need it. +## Provider policy + +The default aggregation should maximize **independent, anonymous coverage** +without turning one user query into a request storm. Disabled engines remain +installed and selectable on demand; `inactive` is reserved for engines that +should disappear completely. + +| Category | Default baseline | On-demand / deferred | +|----------|------------------|----------------------| +| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting | +| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed | +| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested | +| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News | +| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines | +| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines | + +This policy uses no API credentials and does not identify the operator to a +search provider. It is deliberately conservative: a bundled engine is not +necessarily healthy, anonymous, or suitable for every default query. + ## Root cause (one line) All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so @@ -51,6 +71,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**. ## Fix A — Engine config (settings.yml `engines:`) +The current tracked policy enables Bing and DuckDuckGo for default general +search, gives each weight 2, disables pagination, and allows 5 seconds for the +VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained +but disabled from default aggregation. This limits fan-out while preserving +explicit/on-demand selection. + Important: SearXNG's engines are configured in the **`engines:` list inside `settings.yml`**, not a separate `engines.yml` file. With `use_default_settings: true` (our setting) your `engines:` list is *merged* over @@ -322,17 +348,23 @@ misleading). ## Recommended path for Dankclaw (ranked) -1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see +1. **Measure individual engines** — use `/json` + `?engines=` and several + query classes before enabling a bundled engine in default aggregation. +2. **Keep default fan-out small** — prefer independent anonymous providers and + disable pagination for rate-sensitive engines. +3. **Promote by evidence** — raise `weight` only for engines that return useful + results consistently; weight is not a health check. +4. **Avoid credentials** — do not add API-backed engines unless the provider is + genuinely free, anonymous, and compatible with our privacy requirements. +5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see exactly which engines are down and why. Confirm the running SearXNG version (check whether `parsers`/`rate_limit` exist in the pinned build). -2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per - upstream request, caps it at 12 seconds, and retries once. This addresses +6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per + upstream request, caps it at 15 seconds, and retries once. This addresses modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only after measuring the resulting error and latency profile. -3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the - other returning results. -4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a - dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation +7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at + a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation isolation is needed). Avoid the same-netns-proxy trap. --- diff --git a/docs/Dankclaw/SearXNG.md b/docs/Dankclaw/SearXNG.md index 9e760e0..808ac92 100644 --- a/docs/Dankclaw/SearXNG.md +++ b/docs/Dankclaw/SearXNG.md @@ -61,9 +61,29 @@ valkey: url: "valkey://searxng-valkey:6379/0" outgoing: - request_timeout: 4.0 - max_request_timeout: 12.0 + request_timeout: 5.0 + max_request_timeout: 15.0 retries: 1 # avoid amplifying upstream 429/CAPTCHA responses + +# Anonymous, bounded default aggregation. Disabled engines remain configured +# for explicit/on-demand selection; see SearXNG Providers for the policy. +engines: + - name: bing + disabled: false + weight: 2 + paging: false + timeout: 5.0 + - name: duckduckgo + disabled: false + weight: 2 + paging: false + timeout: 5.0 + - name: google cse + disabled: true + - name: brave + disabled: true + - name: startpage + disabled: true ``` ## Volumes @@ -100,9 +120,12 @@ The current request policy allows a modest amount of VPN latency (`4.0s` per request, `12.0s` maximum) while limiting retries to one. This can recover slow providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit -IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429, -and SearXNG **suspends** the offending engine for hours to days, which is what -shows up as "Google went unavailable". This is upstream, not local. +IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429. +The default provider policy therefore favors anonymous Bing and DuckDuckGo, +disables observed CAPTCHA/429 providers from normal aggregation, and disables +pagination to reduce upstream request volume. SearXNG **suspends** offending +engines for hours to days, which is what shows up as "Google went unavailable". +This is upstream, not local. Fixes (cheapest first): tune the `outgoing` block (`request_timeout`, `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a diff --git a/shared/searxng/settings.yml b/shared/searxng/settings.yml index dac8c78..e469f3a 100644 --- a/shared/searxng/settings.yml +++ b/shared/searxng/settings.yml @@ -21,14 +21,94 @@ valkey: url: "valkey://searxng-valkey:6379/0" outgoing: - request_timeout: 4.0 - max_request_timeout: 12.0 + request_timeout: 5.0 + max_request_timeout: 15.0 pool_connections: 100 pool_maxsize: 20 enable_http2: true keepalive_expiry: 5.0 retries: 1 +# Keep default aggregation anonymous and bounded. Disabled engines remain +# available for explicit selection/on-demand use; inactive engines are removed. +# Provider-specific overrides below also reduce pagination fan-out. +engines: + - name: bing + disabled: false + weight: 2 + paging: false + timeout: 5.0 + - name: bing images + paging: false + timeout: 5.0 + - name: bing news + paging: false + timeout: 5.0 + - name: bing videos + paging: false + timeout: 5.0 + - name: duckduckgo + disabled: false + weight: 2 + paging: false + timeout: 5.0 + - name: duckduckgo images + paging: false + timeout: 5.0 + - name: duckduckgo news + paging: false + timeout: 5.0 + - name: duckduckgo videos + paging: false + timeout: 5.0 + - name: wikipedia + weight: 1 + timeout: 5.0 + - name: google cse + disabled: true + paging: false + timeout: 5.0 + - name: google cse images + disabled: true + paging: false + timeout: 5.0 + - name: google news + disabled: true + paging: false + timeout: 5.0 + - name: google scholar + disabled: true + paging: false + timeout: 5.0 + - name: brave + disabled: true + paging: false + timeout: 5.0 + - name: brave.images + disabled: true + paging: false + timeout: 5.0 + - name: brave.news + disabled: true + paging: false + timeout: 5.0 + - name: brave.videos + disabled: true + paging: false + timeout: 5.0 + - name: startpage + disabled: true + paging: false + timeout: 5.0 + - name: startpage images + disabled: true + paging: false + timeout: 5.0 + - name: startpage news + disabled: true + paging: false + timeout: 5.0 + plugins: searx.plugins.unit_converter.SXNGPlugin: active: true diff --git a/tests/searxng_limiter.bats b/tests/searxng_limiter.bats index d104145..0600fd4 100644 --- a/tests/searxng_limiter.bats +++ b/tests/searxng_limiter.bats @@ -63,13 +63,32 @@ import sys, yaml with open(sys.argv[1]) as fh: data = yaml.safe_load(fh) outgoing = data.get("outgoing", {}) -assert outgoing.get("request_timeout") == 4.0, outgoing -assert outgoing.get("max_request_timeout") == 12.0, outgoing +assert outgoing.get("request_timeout") == 5.0, outgoing +assert outgoing.get("max_request_timeout") == 15.0, outgoing assert outgoing.get("retries") == 1, outgoing assert outgoing["request_timeout"] < outgoing["max_request_timeout"], outgoing PYEOF } +@test "SearXNG default provider policy favors anonymous healthy engines" { + python3 - "$REPO_ROOT/shared/searxng/settings.yml" <<'PYEOF' +import sys, yaml +with open(sys.argv[1]) as fh: + data = yaml.safe_load(fh) +engines = {entry["name"]: entry for entry in data.get("engines", [])} +assert engines["bing"]["disabled"] is False, engines.get("bing") +assert engines["bing"]["weight"] > 1, engines["bing"] +assert engines["bing"]["paging"] is False, engines["bing"] +assert engines["duckduckgo"]["disabled"] is False, engines["duckduckgo"] +assert engines["duckduckgo"]["paging"] is False, engines["duckduckgo"] +for name in ("google cse", "startpage", "brave"): + assert engines[name]["disabled"] is True, engines.get(name) + assert engines[name].get("inactive", False) is False, engines[name] +for name in ("bing", "duckduckgo", "wikipedia"): + assert engines[name]["timeout"] == 5.0, engines[name] +PYEOF +} + @test "deploy.sh copies limiter.toml into /opt/searxng/config" { grep -q "shared/searxng/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \ || { echo "deploy.sh does not copy shared/searxng/limiter.toml"; return 1; }