diff --git a/docs/Dankclaw/SearXNG Providers.md b/docs/Dankclaw/SearXNG Providers.md index a159aa6..864407b 100644 --- a/docs/Dankclaw/SearXNG Providers.md +++ b/docs/Dankclaw/SearXNG Providers.md @@ -137,12 +137,12 @@ Tune the `outgoing` block and the suspend model to survive load: ```yaml outgoing: - request_timeout: 6.0 # raise from 2.0 so slow engines aren't dropped as unresponsive - max_request_timeout: 15.0 - pool_connections: 200 # bigger pool = more concurrent connections per engine host - pool_maxsize: 40 # keep-alive connections per host - enable_http2: true # multiplexing = fewer distinct "new connection" events per engine - retries: 3 # on 429/5xx, retry against a different proxy + source IP + request_timeout: 4.0 # current Dankclaw value; modest VPN-latency allowance + max_request_timeout: 12.0 + pool_connections: 100 + pool_maxsize: 20 + enable_http2: true + retries: 1 # avoid amplifying 429/CAPTCHA responses from one exit IP useragent_suffix: "ops@yourhomelab.example" # so a per-engine blocker can reach you search: @@ -159,8 +159,12 @@ Notes: that is a historical-searx concept. The functional equivalent is the suspend schedule above (widen to back off before blocking, narrow to recover faster). - `request_timeout` interacts with the suspend model: too tight → engines look - "unresponsive"; too loose → slow pages. Aim per-engine `timeout` at - *proxy-add-on time + engine latency*, capped by `max_request_timeout`. + "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and + 12 seconds as the cap; raise it only when measurements show real upstream + latency rather than 429/CAPTCHA responses. +- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the + provider's rate limit worse. A retry count of 1 is a bounded compromise for a + transient network failure. - `safe_search: 0` when diagnosing — a too-strict global safe search can make an engine return zero rows and *look* blocked. @@ -288,9 +292,10 @@ misleading). 1. **Verify first** — `/json` + `?engines=google` and the searx-checker to see exactly which engines are down and why. Confirm the running SearXNG version (check whether `parsers`/`rate_limit` exist in the pinned build). -2. **Tighten Fix B** — raise `outgoing.request_timeout`, add `retry_on_http_error: - [403, 429]` on the Google engines, widen `suspended_times`. Cheapest, no new - infra. +2. **Use conservative request tuning** — Dankclaw now allows 4 seconds per + upstream request, caps it at 12 seconds, and retries once. This addresses + modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only + after measuring the resulting error and latency profile. 3. **Duplicate the Google engines** (Fix A) so a CAPTCHA ban on one leaves the other returning results. 4. **If Google still re-blocks** — Fix C Option 1: point the Google engines at a diff --git a/docs/Dankclaw/SearXNG.md b/docs/Dankclaw/SearXNG.md index fbfaedc..9e760e0 100644 --- a/docs/Dankclaw/SearXNG.md +++ b/docs/Dankclaw/SearXNG.md @@ -61,9 +61,9 @@ valkey: url: "valkey://searxng-valkey:6379/0" outgoing: - request_timeout: 3.0 - max_request_timeout: 10.0 - retries: 3 + request_timeout: 4.0 + max_request_timeout: 12.0 + retries: 1 # avoid amplifying upstream 429/CAPTCHA responses ``` ## Volumes @@ -96,8 +96,11 @@ Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local s ## Unblocking Google & Search Providers -SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP. The -rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429, +The current request policy allows a modest amount of VPN latency (`4.0s` per +request, `12.0s` maximum) while limiting retries to one. This can recover slow +providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared +VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit +IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429, and SearXNG **suspends** the offending engine for hours to days, which is what shows up as "Google went unavailable". This is upstream, not local. diff --git a/shared/searxng/settings.yml b/shared/searxng/settings.yml index de8ff0d..dac8c78 100644 --- a/shared/searxng/settings.yml +++ b/shared/searxng/settings.yml @@ -21,13 +21,13 @@ valkey: url: "valkey://searxng-valkey:6379/0" outgoing: - request_timeout: 3.0 - max_request_timeout: 10.0 + request_timeout: 4.0 + max_request_timeout: 12.0 pool_connections: 100 pool_maxsize: 20 enable_http2: true keepalive_expiry: 5.0 - retries: 3 + retries: 1 plugins: searx.plugins.unit_converter.SXNGPlugin: diff --git a/tests/searxng_limiter.bats b/tests/searxng_limiter.bats index f468563..d104145 100644 --- a/tests/searxng_limiter.bats +++ b/tests/searxng_limiter.bats @@ -57,6 +57,19 @@ assert not missing, f"pass_ip missing {missing}; got {pass_ip}" PYEOF } +@test "SearXNG outgoing policy allows modest VPN latency without retry amplification" { + python3 - "$REPO_ROOT/shared/searxng/settings.yml" <<'PYEOF' +import sys, yaml +with open(sys.argv[1]) as fh: + data = yaml.safe_load(fh) +outgoing = data.get("outgoing", {}) +assert outgoing.get("request_timeout") == 4.0, outgoing +assert outgoing.get("max_request_timeout") == 12.0, outgoing +assert outgoing.get("retries") == 1, outgoing +assert outgoing["request_timeout"] < outgoing["max_request_timeout"], outgoing +PYEOF +} + @test "deploy.sh copies limiter.toml into /opt/searxng/config" { grep -q "shared/searxng/limiter.toml" "$REPO_ROOT/scripts/deploy.sh" \ || { echo "deploy.sh does not copy shared/searxng/limiter.toml"; return 1; }