configure anonymous SearXNG provider defaults
This commit is contained in:
Frank
@@ -15,6 +15,26 @@ engine for an hour to 15 days and silently drops it from results. That is the
|
||||
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
|
||||
are the detail if you need it.
|
||||
|
||||
## Provider policy
|
||||
|
||||
The default aggregation should maximize **independent, anonymous coverage**
|
||||
without turning one user query into a request storm. Disabled engines remain
|
||||
installed and selectable on demand; `inactive` is reserved for engines that
|
||||
should disappear completely.
|
||||
|
||||
| Category | Default baseline | On-demand / deferred |
|
||||
|----------|------------------|----------------------|
|
||||
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
|
||||
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
|
||||
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
|
||||
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
|
||||
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
|
||||
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
|
||||
|
||||
This policy uses no API credentials and does not identify the operator to a
|
||||
search provider. It is deliberately conservative: a bundled engine is not
|
||||
necessarily healthy, anonymous, or suitable for every default query.
|
||||
|
||||
## Root cause (one line)
|
||||
|
||||
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
|
||||
@@ -51,6 +71,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
|
||||
|
||||
## Fix A — Engine config (settings.yml `engines:`)
|
||||
|
||||
The current tracked policy enables Bing and DuckDuckGo for default general
|
||||
search, gives each weight 2, disables pagination, and allows 5 seconds for the
|
||||
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
|
||||
but disabled from default aggregation. This limits fan-out while preserving
|
||||
explicit/on-demand selection.
|
||||
|
||||
Important: SearXNG's engines are configured in the **`engines:` list inside
|
||||
`settings.yml`**, not a separate `engines.yml` file. With
|
||||
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
|
||||
@@ -322,17 +348,23 @@ misleading).
|
||||
|
||||
## Recommended path for Dankclaw (ranked)
|
||||
|
||||
1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
|
||||
1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
|
||||
query classes before enabling a bundled engine in default aggregation.
|
||||
2. **Keep default fan-out small** — prefer independent anonymous providers and
|
||||
disable pagination for rate-sensitive engines.
|
||||
3. **Promote by evidence** — raise `weight` only for engines that return useful
|
||||
results consistently; weight is not a health check.
|
||||
4. **Avoid credentials** — do not add API-backed engines unless the provider is
|
||||
genuinely free, anonymous, and compatible with our privacy requirements.
|
||||
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
|
||||
exactly which engines are down and why. Confirm the running SearXNG version
|
||||
(check whether `parsers`/`rate_limit` exist in the pinned build).
|
||||
2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per
|
||||
upstream request, caps it at 12 seconds, and retries once. This addresses
|
||||
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
|
||||
upstream request, caps it at 15 seconds, and retries once. This addresses
|
||||
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
|
||||
after measuring the resulting error and latency profile.
|
||||
3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
|
||||
other returning results.
|
||||
4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
|
||||
dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
|
||||
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
|
||||
a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
|
||||
isolation is needed). Avoid the same-netns-proxy trap.
|
||||
|
||||
---
|
||||
|
||||
@@ -61,9 +61,29 @@ valkey:
|
||||
url: "valkey://searxng-valkey:6379/0"
|
||||
|
||||
outgoing:
|
||||
request_timeout: 4.0
|
||||
max_request_timeout: 12.0
|
||||
request_timeout: 5.0
|
||||
max_request_timeout: 15.0
|
||||
retries: 1 # avoid amplifying upstream 429/CAPTCHA responses
|
||||
|
||||
# Anonymous, bounded default aggregation. Disabled engines remain configured
|
||||
# for explicit/on-demand selection; see SearXNG Providers for the policy.
|
||||
engines:
|
||||
- name: bing
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: duckduckgo
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: google cse
|
||||
disabled: true
|
||||
- name: brave
|
||||
disabled: true
|
||||
- name: startpage
|
||||
disabled: true
|
||||
```
|
||||
|
||||
## Volumes
|
||||
@@ -100,9 +120,12 @@ The current request policy allows a modest amount of VPN latency (`4.0s` per
|
||||
request, `12.0s` maximum) while limiting retries to one. This can recover slow
|
||||
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
|
||||
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
|
||||
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
|
||||
and SearXNG **suspends** the offending engine for hours to days, which is what
|
||||
shows up as "Google went unavailable". This is upstream, not local.
|
||||
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
|
||||
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
|
||||
disables observed CAPTCHA/429 providers from normal aggregation, and disables
|
||||
pagination to reduce upstream request volume. SearXNG **suspends** offending
|
||||
engines for hours to days, which is what shows up as "Google went unavailable".
|
||||
This is upstream, not local.
|
||||
|
||||
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
|
||||
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
|
||||
|
||||
Reference in New Issue
Block a user