SearXNG queries all ~50 upstream engines from one Gluetun VPN exit IP, so
rate-sensitive engines (Google, DuckDuckGo) answer with reCAPTCHA/429 and
SearXNG suspends the offending engine for hours-to-days, showing up as
"unavailable". Add docs/Dankclaw/SearXNG Providers.md covering: the
suspension model, the two failure modes (CAPTCHA vs selector drift), engine
config (settings.yml engines: list — duplicate google, timeout,
retry_on_http_error, JSON feed), request tuning (outgoing block, UA rotation,
suspended_times — correcting the old rate_limit/useragents blog myths),
routing/proxies (per-engine proxies, dedicated 2nd Gluetun, dedicated IP), and
verification (/json, /config, check.searx.space). Crosslink into SearXNG.md
and index.md.
Fanned out 5 read-only doc-auditor subagents (one per service group) that
fact-checked the knowledge base against the live system + repo config,
researched official setup docs online, and verified progress. Applied:
- All 34 broken links fixed (real pages / external URLs / correct relative
paths); 0 broken links remain. No .AGENTTRASH links (the 4 mentions are
accurate descriptive text, not links).
- Fact-check fixes: Readarr retired (May 2026), Radarr v4->v6, Calibre
9.2.1 live, Calibre-Web 0.6.25 live, Copyparty v1.20.21, LLM index
ncmoe 38 + 786432 ctx, frank-default-small active (not frank-pp-worker),
Hermes/SearXNG/Grafana hostnames, Loki/Alloy local images, Observability
diagram (Promtail->Alloy), VLLM retired note, DFlash status, benchmark
live-drift notes, Homepage test count 116.
The JSON rate cap (API_MAX=4/hour) is hardcoded in Python, not settable in
limiter.toml; pass_ip is the only bypass. Rootless podman strips the client IP
on published-port hits (the container sees a slirp source), so local subagents
must present a pass_ip source. Document the slirp IP-strip, the pass_ip
expansion, and the root-owned config file (one-off sudo cp to update).
Update SearXNG, Gluetun, and Valkey to describe the quadlet units instead of
compose: searxng sets Network=gluetun.container, the units install to
~/.config/containers/systemd/, and scripts/deploy-searxng-quadlet.sh deploys
them. Add a Boot Persistence section to the Stack doc. Correct the Gluetun
tunnel check (the :8000 server 404s; verify tun0 instead) and note the pinned
10.89.2.0/24 subnet.
A direct hit on the published port has no X-Forwarded-For header, so the
bot-detection cannot resolve a client and returns 429. SearXNG must be
reached through Caddy (search.dankclaw.local), which sets X-Forwarded-For.
Add the podman bridge subnets to trusted_proxies so Caddy's source is
trusted for XFF extraction; note the CIDR is dynamic and should be pinned.
Verified agentic JSON search returns 26-31 relevant results via this path.
The trusted group lives in limiter.toml, not settings.yml. Add
shared/searxng/limiter.toml trusting 192.168.0.0/24 in pass_ip and
trusted_proxies, copy it into /opt/searxng/config on deploy, and correct
the SearXNG doc. New test seam: tests/searxng_limiter.bats.
Frank