dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

configure anonymous SearXNG provider defaults

Browse Source
This commit is contained in:
Frank
2026-08-25 17:58:22 -07:00
parent e6670f914f
commit 2693259227
4 changed files with 170 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Dankclaw/SearXNG Providers.md
+39 -7
View File
@@ -15,6 +15,26 @@ engine for an hour to 15 days and silently drops it from results. That is the
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
are the detail if you need it.
## Provider policy
The default aggregation should maximize **independent, anonymous coverage**
without turning one user query into a request storm. Disabled engines remain
installed and selectable on demand; `inactive` is reserved for engines that
should disappear completely.
| Category | Default baseline | On-demand / deferred |
|----------|------------------|----------------------|
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
This policy uses no API credentials and does not identify the operator to a
search provider. It is deliberately conservative: a bundled engine is not
necessarily healthy, anonymous, or suitable for every default query.
## Root cause (one line)
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
@@ -51,6 +71,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
## Fix A — Engine config (settings.yml `engines:`)
The current tracked policy enables Bing and DuckDuckGo for default general
search, gives each weight 2, disables pagination, and allows 5 seconds for the
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
but disabled from default aggregation. This limits fan-out while preserving
explicit/on-demand selection.
Important: SearXNG's engines are configured in the **`engines:` list inside
`settings.yml`**, not a separate `engines.yml` file. With
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
@@ -322,17 +348,23 @@ misleading).
## Recommended path for Dankclaw (ranked)
1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
query classes before enabling a bundled engine in default aggregation.
2. **Keep default fan-out small** — prefer independent anonymous providers and
disable pagination for rate-sensitive engines.
3. **Promote by evidence** — raise `weight` only for engines that return useful
results consistently; weight is not a health check.
4. **Avoid credentials** — do not add API-backed engines unless the provider is
genuinely free, anonymous, and compatible with our privacy requirements.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per
upstream request, caps it at 12 seconds, and retries once. This addresses
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
upstream request, caps it at 15 seconds, and retries once. This addresses
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
after measuring the resulting error and latency profile.
3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the
other returning results.
4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a
dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
isolation is needed). Avoid the same-netns-proxy trap.
---
docs/Dankclaw/SearXNG.md
+28 -5
View File
@@ -61,9 +61,29 @@ valkey:
url: "valkey://searxng-valkey:6379/0"
outgoing:
request_timeout: 4.0
max_request_timeout: 12.0
request_timeout: 5.0
max_request_timeout: 15.0
retries: 1 # avoid amplifying upstream 429/CAPTCHA responses
# Anonymous, bounded default aggregation. Disabled engines remain configured
# for explicit/on-demand selection; see SearXNG Providers for the policy.
engines:
- name: bing
disabled: false
weight: 2
paging: false
timeout: 5.0
- name: duckduckgo
disabled: false
weight: 2
paging: false
timeout: 5.0
- name: google cse
disabled: true
- name: brave
disabled: true
- name: startpage
disabled: true
```
## Volumes
@@ -100,9 +120,12 @@ The current request policy allows a modest amount of VPN latency (`4.0s` per
request, `12.0s` maximum) while limiting retries to one. This can recover slow
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429,
and SearXNG **suspends** the offending engine for hours to days, which is what
shows up as "Google went unavailable". This is upstream, not local.
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
disables observed CAPTCHA/429 providers from normal aggregation, and disables
pagination to reduce upstream request volume. SearXNG **suspends** offending
engines for hours to days, which is what shows up as "Google went unavailable".
This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
shared/searxng/settings.yml
+82 -2
View File
@@ -21,14 +21,94 @@ valkey:
url: "valkey://searxng-valkey:6379/0"
outgoing:
request_timeout: 4.0
max_request_timeout: 12.0
request_timeout: 5.0
max_request_timeout: 15.0
pool_connections: 100
pool_maxsize: 20
enable_http2: true
keepalive_expiry: 5.0
retries: 1
# Keep default aggregation anonymous and bounded. Disabled engines remain
# available for explicit selection/on-demand use; inactive engines are removed.
# Provider-specific overrides below also reduce pagination fan-out.
engines:
- name: bing
disabled: false
weight: 2
paging: false
timeout: 5.0
- name: bing images
paging: false
timeout: 5.0
- name: bing news
paging: false
timeout: 5.0
- name: bing videos
paging: false
timeout: 5.0
- name: duckduckgo
disabled: false
weight: 2
paging: false
timeout: 5.0
- name: duckduckgo images
paging: false
timeout: 5.0
- name: duckduckgo news
paging: false
timeout: 5.0
- name: duckduckgo videos
paging: false
timeout: 5.0
- name: wikipedia
weight: 1
timeout: 5.0
- name: google cse
disabled: true
paging: false
timeout: 5.0
- name: google cse images
disabled: true
paging: false
timeout: 5.0
- name: google news
disabled: true
paging: false
timeout: 5.0
- name: google scholar
disabled: true
paging: false
timeout: 5.0
- name: brave
disabled: true
paging: false
timeout: 5.0
- name: brave.images
disabled: true
paging: false
timeout: 5.0
- name: brave.news
disabled: true
paging: false
timeout: 5.0
- name: brave.videos
disabled: true
paging: false
timeout: 5.0
- name: startpage
disabled: true
paging: false
timeout: 5.0
- name: startpage images
disabled: true
paging: false
timeout: 5.0
- name: startpage news
disabled: true
paging: false
timeout: 5.0
plugins:
searx.plugins.unit_converter.SXNGPlugin:
active: true
tests/searxng_limiter.bats
+21 -2
View File
@@ -63,13 +63,32 @@ import sys, yaml
with open(sys.argv[1]) as fh:
data = yaml.safe_load(fh)
outgoing = data.get("outgoing", {})
assert outgoing.get("request_timeout") == 4.0, outgoing
assert outgoing.get("max_request_timeout") == 12.0, outgoing
assert outgoing.get("request_timeout") == 5.0, outgoing
assert outgoing.get("max_request_timeout") == 15.0, outgoing
assert outgoing.get("retries") == 1, outgoing
assert outgoing["request_timeout"] < outgoing["max_request_timeout"], outgoing
PYEOF
}
@test "SearXNG default provider policy favors anonymous healthy engines" {
python3 - "$REPO_ROOT/shared/searxng/settings.yml" <<'PYEOF'
import sys, yaml
with open(sys.argv[1]) as fh:
data = yaml.safe_load(fh)
engines = {entry["name"]: entry for entry in data.get("engines", [])}
assert engines["bing"]["disabled"] is False, engines.get("bing")
assert engines["bing"]["weight"] > 1, engines["bing"]
assert engines["bing"]["paging"] is False, engines["bing"]
assert engines["duckduckgo"]["disabled"] is False, engines["duckduckgo"]
assert engines["duckduckgo"]["paging"] is False, engines["duckduckgo"]
for name in ("google cse", "startpage", "brave"):
assert engines[name]["disabled"] is True, engines.get(name)
assert engines[name].get("inactive", False) is False, engines[name]
for name in ("bing", "duckduckgo", "wikipedia"):
assert engines[name]["timeout"] == 5.0, engines[name]
PYEOF
}
@test "deploy.sh copies limiter.toml into /opt/searxng/config" {
grep -q "shared/searxng/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \
|| { echo "deploy.sh does not copy shared/searxng/limiter.toml"; return 1; }