configure anonymous SearXNG provider defaults

docs/Dankclaw/SearXNG Providers.md

@@ -15,6 +15,26 @@ engine for an hour to 15 days and silently drops it from results. That is the
 This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
 are the detail if you need it.
 
+## Provider policy
+
+The default aggregation should maximize **independent, anonymous coverage**
+without turning one user query into a request storm. Disabled engines remain
+installed and selectable on demand; `inactive` is reserved for engines that
+should disappear completely.
+
+| Category | Default baseline | On-demand / deferred |
+|----------|------------------|----------------------|
+| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
+| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
+| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
+| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
+| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
+| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
+
+This policy uses no API credentials and does not identify the operator to a
+search provider. It is deliberately conservative: a bundled engine is not
+necessarily healthy, anonymous, or suitable for every default query.
+
 ## Root cause (one line)
 
 All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
@@ -51,6 +71,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
 
 ## Fix A — Engine config (settings.yml `engines:`)
 
+The current tracked policy enables Bing and DuckDuckGo for default general
+search, gives each weight 2, disables pagination, and allows 5 seconds for the
+VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
+but disabled from default aggregation. This limits fan-out while preserving
+explicit/on-demand selection.
+
 Important: SearXNG's engines are configured in the **`engines:` list inside
 `settings.yml`**, not a separate `engines.yml` file. With
 `use_default_settings: true` (our setting) your `engines:` list is *merged* over
@@ -322,17 +348,23 @@ misleading).
 
 ## Recommended path for Dankclaw (ranked)
 
-1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
+1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
+   query classes before enabling a bundled engine in default aggregation.
+2. **Keep default fan-out small** — prefer independent anonymous providers and
+   disable pagination for rate-sensitive engines.
+3. **Promote by evidence** — raise `weight` only for engines that return useful
+   results consistently; weight is not a health check.
+4. **Avoid credentials** — do not add API-backed engines unless the provider is
+   genuinely free, anonymous, and compatible with our privacy requirements.
+5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
    exactly which engines are down and why. Confirm the running SearXNG version
    (check whether `parsers`/`rate_limit` exist in the pinned build).
-2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per
+6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
-   upstream request, caps it at 12 seconds, and retries once. This addresses
+   upstream request, caps it at 15 seconds, and retries once. This addresses
    modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
    after measuring the resulting error and latency profile.
-3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
+7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
-   other returning results.
+   a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
-4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
-   dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
    isolation is needed). Avoid the same-netns-proxy trap.
 
 ---

docs/Dankclaw/SearXNG.md

@@ -61,9 +61,29 @@ valkey:
   url: "valkey://searxng-valkey:6379/0"
 
 outgoing:
-  request_timeout: 4.0
+  request_timeout: 5.0
-  max_request_timeout: 12.0
+  max_request_timeout: 15.0
   retries: 1                     # avoid amplifying upstream 429/CAPTCHA responses
+
+# Anonymous, bounded default aggregation. Disabled engines remain configured
+# for explicit/on-demand selection; see SearXNG Providers for the policy.
+engines:
+  - name: bing
+    disabled: false
+    weight: 2
+    paging: false
+    timeout: 5.0
+  - name: duckduckgo
+    disabled: false
+    weight: 2
+    paging: false
+    timeout: 5.0
+  - name: google cse
+    disabled: true
+  - name: brave
+    disabled: true
+  - name: startpage
+    disabled: true
 ```
 
 ## Volumes
@@ -100,9 +120,12 @@ The current request policy allows a modest amount of VPN latency (`4.0s` per
 request, `12.0s` maximum) while limiting retries to one. This can recover slow
 providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
 VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
-IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
+IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
-and SearXNG **suspends** the offending engine for hours to days, which is what
+The default provider policy therefore favors anonymous Bing and DuckDuckGo,
-shows up as "Google went unavailable". This is upstream, not local.
+disables observed CAPTCHA/429 providers from normal aggregation, and disables
+pagination to reduce upstream request volume. SearXNG **suspends** offending
+engines for hours to days, which is what shows up as "Google went unavailable".
+This is upstream, not local.
 
 Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
 `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a

shared/searxng/settings.yml

@@ -21,14 +21,94 @@ valkey:
   url: "valkey://searxng-valkey:6379/0"
 
 outgoing:
-  request_timeout: 4.0
+  request_timeout: 5.0
-  max_request_timeout: 12.0
+  max_request_timeout: 15.0
   pool_connections: 100
   pool_maxsize: 20
   enable_http2: true
   keepalive_expiry: 5.0
   retries: 1
 
+# Keep default aggregation anonymous and bounded. Disabled engines remain
+# available for explicit selection/on-demand use; inactive engines are removed.
+# Provider-specific overrides below also reduce pagination fan-out.
+engines:
+  - name: bing
+    disabled: false
+    weight: 2
+    paging: false
+    timeout: 5.0
+  - name: bing images
+    paging: false
+    timeout: 5.0
+  - name: bing news
+    paging: false
+    timeout: 5.0
+  - name: bing videos
+    paging: false
+    timeout: 5.0
+  - name: duckduckgo
+    disabled: false
+    weight: 2
+    paging: false
+    timeout: 5.0
+  - name: duckduckgo images
+    paging: false
+    timeout: 5.0
+  - name: duckduckgo news
+    paging: false
+    timeout: 5.0
+  - name: duckduckgo videos
+    paging: false
+    timeout: 5.0
+  - name: wikipedia
+    weight: 1
+    timeout: 5.0
+  - name: google cse
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: google cse images
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: google news
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: google scholar
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: brave
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: brave.images
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: brave.news
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: brave.videos
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: startpage
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: startpage images
+    disabled: true
+    paging: false
+    timeout: 5.0
+  - name: startpage news
+    disabled: true
+    paging: false
+    timeout: 5.0
+
 plugins:
   searx.plugins.unit_converter.SXNGPlugin:
     active: true

tests/searxng_limiter.bats

@@ -63,13 +63,32 @@ import sys, yaml
 with open(sys.argv[1]) as fh:
     data = yaml.safe_load(fh)
 outgoing = data.get("outgoing", {})
-assert outgoing.get("request_timeout") == 4.0, outgoing
+assert outgoing.get("request_timeout") == 5.0, outgoing
-assert outgoing.get("max_request_timeout") == 12.0, outgoing
+assert outgoing.get("max_request_timeout") == 15.0, outgoing
 assert outgoing.get("retries") == 1, outgoing
 assert outgoing["request_timeout"] < outgoing["max_request_timeout"], outgoing
 PYEOF
 }
 
+@test "SearXNG default provider policy favors anonymous healthy engines" {
+    python3 - "$REPO_ROOT/shared/searxng/settings.yml" <<'PYEOF'
+import sys, yaml
+with open(sys.argv[1]) as fh:
+    data = yaml.safe_load(fh)
+engines = {entry["name"]: entry for entry in data.get("engines", [])}
+assert engines["bing"]["disabled"] is False, engines.get("bing")
+assert engines["bing"]["weight"] > 1, engines["bing"]
+assert engines["bing"]["paging"] is False, engines["bing"]
+assert engines["duckduckgo"]["disabled"] is False, engines["duckduckgo"]
+assert engines["duckduckgo"]["paging"] is False, engines["duckduckgo"]
+for name in ("google cse", "startpage", "brave"):
+    assert engines[name]["disabled"] is True, engines.get(name)
+    assert engines[name].get("inactive", False) is False, engines[name]
+for name in ("bing", "duckduckgo", "wikipedia"):
+    assert engines[name]["timeout"] == 5.0, engines[name]
+PYEOF
+}
+
 @test "deploy.sh copies limiter.toml into /opt/searxng/config" {
     grep -q "shared/searxng/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \
         || { echo "deploy.sh does not copy shared/searxng/limiter.toml"; return 1; }