docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
@@ -14,12 +14,15 @@ VPN tunnel, and how to unblock them in SearXNG.

 **Host:** Dankclaw — **Docs:** https://docs.searxng.org

-SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
-Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
+SearXNG aggregates the engines bundled in the running build — **262 installed, 75
+enabled** on the live instance (`GET http://127.0.0.1:8081/config`, 2026-10-18; the
+"~50 engines" framing this page carried predates that count). When all of them are
+queried through **one** VPN exit IP — as on
 Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
 start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
-engine for an hour to 15 days and silently drops it from results. That is the
-"Google (and sometimes a few others) went unavailable" symptom.
+engine (three minutes for a 429 under the current build's defaults, up to 15 days for
+a Cloudflare CAPTCHA — see the table below) and silently drops it from results. That
+is the "Google (and sometimes a few others) went unavailable" symptom.

 This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
 are the detail if you need it.
@@ -35,20 +38,32 @@ should disappear completely.
 |----------|------------------|----------------------|
 | General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
 | Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
-| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
+| Science | arXiv, PubMed, Semantic Scholar | Google Scholar until independently tested; OpenAlex (`openalex` is `enabled: false` in the running build, `/config` 2026-10-18) |
 | News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
-| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
+| Images | Wikimedia Commons (ships as `wikicommons.images/videos/audio/files`, all enabled), Bing Images, Unsplash | Openverse (`enabled: false` today), Google/Brave/Startpage image engines |
 | Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |

-This policy uses no API credentials and does not identify the operator to a
-search provider. It is deliberately conservative: a bundled engine is not
+**Live state of this table (2026-10-18, from `GET /config`):** the General/web,
+Technical, News and Videos baselines hold engine-for-engine; the two rows above are
+the corrections. Nothing in the deferred column is enabled. One operational note for
+the deferred engines: re-enabling Google CSE on demand needs a key this host does not
+hold — `/opt/searxng/.env` carries only the WireGuard variables plus
+`SEARXNG_SECRET`, `SEARXNG_BASE_URL` and `KAGI_API_KEY` (key **names** read
+2026-10-18; values not printed).
+
+This policy was written to use no API credentials and to avoid identifying the
+operator to a search provider — **with one carve-out made afterwards (2026-09-26):
+the paid `kagi` engine, whose key is rendered into the deployed config from
+`/opt/searxng/.env` (`shared/searxng/settings.yml:70`; item 4 below and
+[[SearXNG]]).** It is deliberately conservative: a bundled engine is not
 necessarily healthy, anonymous, or suitable for every default query.

 ## Root cause (one line)

 All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
-Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
-rate-limit it → CAPTCHA → engine **suspension** → empty results.
+Google/DuckDuckGo see one datacenter IP spamming every enabled engine (75 of the 262
+installed ones, `/config` 2026-10-18) and rate-limit it → CAPTCHA → engine
+**suspension** → empty results.

 ## The suspension model (what "unavailable" actually is)

@@ -58,15 +73,23 @@ Suspended engines are the usual cause of "gone":

 | Exception | Default suspension | Trigger |
 |-----------|--------------------|---------|
-| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
-| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
-| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
-| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
-| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
-| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
+| `SearxEngineTooManyRequests` | **180 s (3 min)** | HTTP 429 |
+| `SearxEngineAccessDenied` | **180 s (3 min)** | 403 / access denied |
+| `SearxEngineCaptcha` | **3600 s (1 h)** | Google `/sorry/` or reCAPTCHA |
+| `recaptcha_SearxEngineCaptcha` | ~7 days (604 800 s) | Google reCAPTCHA |
+| `cf_SearxEngineCaptcha` | ~15 days (1 296 000 s) | Cloudflare edge CAPTCHA |
+| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail` 5 / `max_ban_time_on_fail` 120 |

-The values are `search.suspended_times` (+ `ban_time_on_fail`) in
-`settings.yml`. An engine that returns **0 results without an error** is *not*
+These are the defaults of the **running commit** `12f8b6515`, read from
+[`searx/settings.yml` at that commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml)
+(lines 66-81). The folklore numbers this table used to carry — ~1 hour for 429, ~1 day
+for access-denied and CAPTCHA — were the **older** upstream defaults and are retired;
+the current build un-suspends a 429'd engine after three minutes.
+
+The values are `search.suspended_times` (+ `ban_time_on_fail`). Our
+`shared/searxng/settings.yml` overrides **none** of them (`grep -n "suspended_times\|ban_time_on_fail"`
+returns nothing, 2026-10-18), so the upstream defaults quoted above are exactly what
+runs. An engine that returns **0 results without an error** is *not*
 suspended — that is the separate "selector drift" problem (below).

 ## Two failure modes
@@ -81,9 +104,12 @@ Most "Google is broken on a VPN host" cases are a **mix of both**.
 ## Fix A — Engine config (settings.yml `engines:`)

 The current tracked policy enables Bing and DuckDuckGo for default general
-search, gives each weight 2, disables pagination, and allows 5 seconds for the
-VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
-but disabled from default aggregation. This limits fan-out while preserving
+search, gives each weight 2, disables pagination, and allows 5.0 seconds for the
+VPN path (`shared/searxng/settings.yml:26,38-56`). CAPTCHA/429-prone Google, Brave,
+and Startpage families are retained but disabled from default aggregation
+(`:81-124`) — though the three `startpage*` stanzas are dead weight: the running
+build ships no `startpage` engine at all (`/config` lists none, 2026-10-18), so those
+overrides match nothing. This limits fan-out while preserving
 explicit/on-demand selection.

 Important: SearXNG's engines are configured in the **`engines:` list inside
@@ -105,7 +131,7 @@ engines:
  - name: google
    weight: 3
    disabled: false
-    timeout: 5.0                 # wait longer over the tunnel (global default is 2.0s)
+    timeout: 5.0                 # wait longer over the tunnel (upstream global default is 3.0s — searx/settings.yml:178 at the running commit)
    retry_on_http_error: [403, 429]
    display_error_messages: true

@@ -172,8 +198,8 @@ Tune the `outgoing` block and the suspend model to survive load:

 ```yaml
 outgoing:
-  request_timeout: 4.0             # current Dankclaw value; modest VPN-latency allowance
-  max_request_timeout: 12.0
+  request_timeout: 5.0             # Dankclaw's value (shared/searxng/settings.yml:26)
+  max_request_timeout: 15.0        # Dankclaw's cap (shared/searxng/settings.yml:27)
  pool_connections: 100
  pool_maxsize: 20
  enable_http2: true
@@ -194,9 +220,9 @@ Notes:
  that is a historical-searx concept. The functional equivalent is the suspend
  schedule above (widen to back off before blocking, narrow to recover faster).
 - `request_timeout` interacts with the suspend model: too tight → engines look
-  "unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
-  12 seconds as the cap; raise it only when measurements show real upstream
-  latency rather than 429/CAPTCHA responses.
+  "unresponsive"; too loose → slow pages. Dankclaw uses 5 seconds globally and
+  15 seconds as the cap (`shared/searxng/settings.yml:26-27`); raise it only when
+  measurements show real upstream latency rather than 429/CAPTCHA responses.
 - Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
  provider's rate limit worse. A retry count of 1 is a bounded compromise for a
  transient network failure.
@@ -281,7 +307,14 @@ The read-only audit found:
  Google's "unusual traffic from your network" response.
 - `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
  healthy. This isolates the primary failures to upstream provider handling,
-  not application availability.
+  not application availability. Re-checked 2026-10-18: `/healthz` is still `200`
+  and `gluetun`/`searxng`/`searxng-valkey` are all `generated` + `active`.
+
+What has moved since that audit: the container is still
+`ghcr.io/searxng/searxng:latest` (`podman inspect searxng` 2026-10-18), but the
+embedded version is now `2026.9.25+12f8b6515` (`GET /config`), not `1.0.0`, and
+`limiter.enabled` is `true`. The exit IP was not re-measured (that would need a
+query fired through the tunnel).

 This deployment returns the legacy array form for `unresponsive_engines`:

@@ -293,10 +326,15 @@ This deployment returns the legacy array form for `unresponsive_engines`:
 ```

 Do not assume the newer object form (`engine`, `error_type`, `suspended`)
-until the live response has been checked. The direct published port is the
-useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
-hostname resolved to the WAN address during the audit and timed out from the
-audit host, so it was not used as the baseline path.
+until the live response has been checked — and note that the jq recipes below key on
+`.engine`, which only matches the object form; against the legacy `[engine, message]`
+pair they silently match nothing. *(Re-checked 2026-10-18 with a `bing`-only JSON
+query: `unresponsive_engines` came back empty, so today's shape is unverified.)* The
+direct published port is the useful diagnostic path because it exercises the
+ProtonVPN exit. The Caddy hostname resolved to the WAN address during the audit and
+timed out from the audit host, so it was not used as the baseline path; on
+2026-10-18 `search.franksdank.cloud` resolves to the LAN IP from inside the network
+and answers with the Anubis challenge page.

 ## Fix D — Verify an engine is unblocked

@@ -304,7 +342,9 @@ Three layers, cheapest first. All queries should hit the path production uses
 (the **published Gluetun port → ProtonVPN exit**, not a direct host port).

 **1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
-(otherwise `format=json` → 403), then query a single engine:
+(otherwise `format=json` → 403) — `html` and `json` are both enabled here
+(`shared/searxng/settings.yml:18-20`), and a JSON query through the published port
+returns 200 `application/json` today (2026-10-18). Query a single engine:

 ```bash
 BASE="http://192.168.0.143:8081"   # published port (ProtonVPN exit)
@@ -335,7 +375,10 @@ done
 ```

 **2. `/config` REST endpoint** lists every engine + enabled/disabled +
-categories — an inventory of what is live. `?engines=<name>` isolates a single
+categories — an inventory of what is live. It answers 200 on this instance today
+(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
+`categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
+[[SearXNG]]). `?engines=<name>` isolates a single
 provider; `/debug` adds engine diagnostics.

 **3. `check.searx.space`** — the hosted official checker probes every engine of
@@ -369,14 +412,17 @@ misleading).
   engine (Kagi) is acceptable when the blast radius is bounded — the
   `kagi` engine runs with `paging: false` and weight 2, so it spends exactly
   one API credit per aggregated query, and its key never enters git
-   (deploy-time rendering from `/opt/searxng/.env`; see [[SearXNG]]). It is
+   (deploy-time rendering from `/opt/searxng/.env` into
+   `/opt/searxng/config/settings.yml`, where the `${KAGI_API_KEY}` placeholder is
+   already replaced — checked 2026-10-18; see [[SearXNG]]). It is
   also immune to the shared-VPN-exit-IP blocking this page is about: it
   authenticates, so reputation of the exit IP is irrelevant.
 5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
   exactly which engines are down and why. Confirm the running SearXNG version
   (check whether `parsers`/`rate_limit` exist in the pinned build).
 6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
-   upstream request, caps it at 15 seconds, and retries once. This addresses
+   upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
+   (`:27`), and retries once (`:32`). This addresses
   modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
   after measuring the resulting error and latency profile.
 7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
@@ -407,6 +453,8 @@ misleading).
 - [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
 - [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
 - [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
+- [Upstream `searx/settings.yml` at the running commit `12f8b6515`](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/settings.yml) — the `suspended_times`, `ban_time_on_fail` and `request_timeout` defaults quoted above.
+- [Upstream `searx/engines/kagi.py` at the same commit](https://raw.githubusercontent.com/searxng/searxng/12f8b6515/searx/engines/kagi.py) — Bearer auth, `POST https://kagi.com/api/v1/search`, `require_api_key: True`.

 ## See Also