searxng: wire paid Kagi engine with deploy-time key rendering

The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).

- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
  exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
  so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
  to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
  on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
  /opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
  into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
  the no-credentials policy (paid engine acceptable when bounded + ordered).

Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
This commit is contained in:
Frank
2026-09-26 23:25:55 -07:00
parent 4992e5a010
commit 7a8d992fce
7 changed files with 198 additions and 8 deletions
@@ -363,8 +363,15 @@ misleading).
disable pagination for rate-sensitive engines.
3. **Promote by evidence** — raise `weight` only for engines that return useful
results consistently; weight is not a health check.
4. **Avoid credentials** — do not add API-backed engines unless the provider is
genuinely free, anonymous, and compatible with our privacy requirements.
4. **Avoid credentials, with one carve-out** — do not add API-backed engines
unless the provider is genuinely free, anonymous, and compatible with our
privacy requirements. Carve-out (2026-09-26): a paid, user-ordered API
engine (Kagi) is acceptable when the blast radius is bounded — the
`kagi` engine runs with `paging: false` and weight 2, so it spends exactly
one API credit per aggregated query, and its key never enters git
(deploy-time rendering from `/opt/searxng/.env`; see [[SearXNG]]). It is
also immune to the shared-VPN-exit-IP blocking this page is about: it
authenticates, so reputation of the exit IP is irrelevant.
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
exactly which engines are down and why. Confirm the running SearXNG version
(check whether `parsers`/`rate_limit` exist in the pinned build).
+42 -5
View File
@@ -60,8 +60,8 @@ Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, d
```yaml
use_default_settings: true
server:
secret_key: "${SEARXNG_SECRET}" # from .env
base_url: "${SEARXNG_BASE_URL}" # resolves to http://192.168.0.143:8081/
secret_key: "${SEARXNG_SECRET}" # rendered at deploy by scripts/searxng-render-settings.py
base_url: "${SEARXNG_BASE_URL}" # same renderer; SearXNG does NOT expand env vars itself
limiter: true
method: "POST"
@@ -90,8 +90,16 @@ engines:
- name: duckduckgo
disabled: false
weight: 2
paging: false
timeout: 5.0
- name: kagi
engine: kagi
categories: [general, web]
shortcut: kg
api_key: "${KAGI_API_KEY}" # rendered at deploy from /opt/searxng/.env
kagi_categ: search
disabled: false
weight: 2
paging: false # one API credit per aggregated query
timeout: 8.0
- name: google cse
disabled: true
- name: brave
@@ -168,10 +176,39 @@ root cause, the suspend-model table, the `engines:` config, and the
LAN by construction (no public route to the LAN IP). No second agent-only
instance is needed; a duplicate would double upstream request volume from the
same rate-sensitive VPN exit IP for zero benefit.
- The config file is root-owned (uid of the rootless podman namespace). Update it with a one-off `sudo cp`, then run `systemctl --user restart searxng.service` to reload.
- The config file is root-owned (uid of the rootless podman namespace). The
official path is `deploy.sh --service searxng`: it copies the repo settings
and renders the `${VAR}` placeholders via `scripts/searxng-render-settings.py`
from `/opt/searxng/.env`. A hotfix can still hand-copy with `sudo cp`, but
remember the renderer: an unrendered copy silently serves the literal
placeholders (before 2026-09-26 the live `secret_key` literally was
`"${SEARXNG_SECRET}"` and nobody noticed).
- `base_url` must match the public address or proxy rewrites break.
- Keep the valkey cache container on the same podman network as searxng.
## Kagi (paid API engine)
Added 2026-09-26 (user-ordered). The stock SearXNG image ships a native
`kagi` engine module (`searx/engines/kagi.py`); it authenticates with
`Authorization: Bearer <key>` against `POST https://kagi.com/api/v1/search`.
- **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
`[general, web]`, shortcut `kg`, `kagi_categ: search`.
- **Cost control**: `paging: false` — exactly one API credit per aggregated
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0).
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
`deploy-remote.sh` bakes the real key into the live settings.yml via
`scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
(dank-owned, mode 600). The key never enters git and never appears in any
process argv. SearXNG itself does NOT expand env vars in settings.yml —
without the renderer the engine fails `setup()` and is dropped at load.
- **Immune to the VPN-exit-IP problem**: Kagi authenticates, so it does not
care about the shared Gluetun exit IP reputation (see [[SearXNG Providers]]).
- **Verified live 2026-09-26**: direct API probe through the gluetun netns
(200, `data.search[]` shape), then a live aggregated query returned
29/29 results tagged `kagi`, zero unresponsive engines.
## See Also
- [[Dankclaw Stack]]