configure anonymous SearXNG provider defaults

docs/Dankclaw/SearXNG Providers.md

@@ -15,6 +15,26 @@ engine for an hour to 15 days and silently drops it from results. That is the
 This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
 are the detail if you need it.
 
+## Provider policy
+
+The default aggregation should maximize **independent, anonymous coverage**
+without turning one user query into a request storm. Disabled engines remain
+installed and selectable on demand; `inactive` is reserved for engines that
+should disappear completely.
+
+| Category | Default baseline | On-demand / deferred |
+|----------|------------------|----------------------|
+| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
+| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
+| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
+| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
+| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
+| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
+
+This policy uses no API credentials and does not identify the operator to a
+search provider. It is deliberately conservative: a bundled engine is not
+necessarily healthy, anonymous, or suitable for every default query.
+
 ## Root cause (one line)
 
 All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
@@ -51,6 +71,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
 
 ## Fix A — Engine config (settings.yml `engines:`)
 
+The current tracked policy enables Bing and DuckDuckGo for default general
+search, gives each weight 2, disables pagination, and allows 5 seconds for the
+VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
+but disabled from default aggregation. This limits fan-out while preserving
+explicit/on-demand selection.
+
 Important: SearXNG's engines are configured in the **`engines:` list inside
 `settings.yml`**, not a separate `engines.yml` file. With
 `use_default_settings: true` (our setting) your `engines:` list is *merged* over
@@ -322,17 +348,23 @@ misleading).
 
 ## Recommended path for Dankclaw (ranked)
 
-1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
+1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
+   query classes before enabling a bundled engine in default aggregation.
+2. **Keep default fan-out small** — prefer independent anonymous providers and
+   disable pagination for rate-sensitive engines.
+3. **Promote by evidence** — raise `weight` only for engines that return useful
+   results consistently; weight is not a health check.
+4. **Avoid credentials** — do not add API-backed engines unless the provider is
+   genuinely free, anonymous, and compatible with our privacy requirements.
+5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
    exactly which engines are down and why. Confirm the running SearXNG version
    (check whether `parsers`/`rate_limit` exist in the pinned build).
-2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per
-   upstream request, caps it at 12 seconds, and retries once. This addresses
+6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
+   upstream request, caps it at 15 seconds, and retries once. This addresses
    modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
    after measuring the resulting error and latency profile.
-3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
-   other returning results.
-4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
-   dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
+7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
+   a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
    isolation is needed). Avoid the same-netns-proxy trap.
 
 ---