dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Dankclaw/SearXNG.md
+121 -48
View File
@@ -17,11 +17,11 @@ SearXNG is a privacy-respecting metasearch engine. It aggregates results from ma
| Item | Value |
|------|-------|
| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
| Port | 8081 (host, on Gluetun) → 8080 (container) |
| Config | `/etc/searxng/settings.yml` (bind mount) |
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
| Formats | `html`, `json` (agent API) |
| Image | `ghcr.io/searxng/searxng:latest` (`dankclaw/searxng/quadlet/searxng.container:19`; the live container's image reference is exactly that — `podman inspect searxng` 2026-10-18). Running build: `2026.9.25+12f8b6515` (`GET /config`) |
| Port | SearXNG listens on **8080 inside Gluetun's network namespace**; the host sees **8081**, published by the gluetun unit (`dankclaw/searxng/quadlet/gluetun.container:31`; `ss -ltn` shows `*:8081`, `curl http://127.0.0.1:8081/` → 200, 2026-10-18) |
| Config | `/etc/searxng/settings.yml` (bind mount of `/opt/searxng/config`, `searxng.container:22`) |
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine`, container `searxng-valkey`, publishes nothing (see [[Valkey]]) |
| Formats | `html`, `json` (agent API) — `shared/searxng/settings.yml:18-20`; `curl "http://127.0.0.1:8081/search?q=…&format=json"` → 200 `application/json` (2026-10-18) |
## VPN Routing (critical)
@@ -37,15 +37,15 @@ Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]]
## Boot Persistence (Quadlet)
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `dankclaw/searxng/quadlet/` and install to `~/.config/containers/systemd/`.
The units live in `dankclaw/searxng/quadlet/` and install to
`~/.config/containers/systemd/` on Dankclaw (verified live):
| Unit file | Service | Role |
|-----------|---------|------|
| `searxng.network` | generated network service | DNS-enabled bridge; the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 |
| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` — the only published port in the trio |
| `searxng.network` | generated network service | DNS-enabled bridge (`dns_enabled: true`, `podman network inspect systemd-searxng` 2026-10-18); the created network is named `systemd-searxng`, subnet pinned to 10.89.2.0/24 (`searxng.network:16`) |
| `gluetun.container` | `gluetun.service` | WireGuard tunnel (env-file secrets); publishes `8081:8080` for SearXNG **and** `8082:8085` for [[Marreta]] (`gluetun.container:31-32`) — SearXNG's is the only port in the search trio, but the tunnel unit carries two, and both are live (`ss -ltn` 2026-10-18) |
| `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache on the searxng network, alias `valkey` (see [[Valkey]]) |
| `searxng.container` | `searxng.service` | `Network=gluetun.container` — joins the tunnel's netns, so it publishes no port of its own; reached through gluetun's 8081 |
@@ -53,7 +53,7 @@ gluetun and valkey join the searxng network. searxng reuses gluetun's netns
and reaches valkey by name on that network. Each unit sets
`WantedBy=default.target`. User linger is on, so the units start at boot.
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw` — a wrapper that execs `scripts/deploy.sh dankclaw --service searxng` (`scripts/deploy-searxng-quadlet.sh:11`). The dispatcher copies the config, installs the three units plus the network, daemon-reloads, and starts them in dependency order: network → gluetun → valkey → searxng (`scripts/deploy-remote.sh:583-593`).
## settings.yml Keys
@@ -108,51 +108,66 @@ engines:
disabled: true
```
The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
`secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
censused below.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/searxng` | `/opt/searxng/config` |
| `/etc/searxng/valkey` | (valkey container data) |
| Container path | Host path | Proof |
|----------------|-----------|-------|
| `/etc/searxng` (searxng) | `/opt/searxng/config` | `searxng.container:22`; live dir holds `settings.yml` + `limiter.toml` (`ls -la /opt/searxng/config` 2026-10-18) |
| `/data` (valkey container) | `/opt/searxng/valkey` | `dankclaw/searxng/quadlet/searxng-valkey.container:20` — this is a mount of the **valkey** container, not of `/etc/searxng/valkey` |
## Healthcheck
SearXNG responds on `/healthz` with 200 when ready.
SearXNG responds on `/healthz` with 200 when ready — re-checked on 2026-10-18:
`curl -s -o /dev/null -w '%{http_code} %{content_type}' http://127.0.0.1:8081/healthz`
→ `200 text/plain`. The unit itself carries no `HealthCmd`, so `podman ps` shows
`searxng` as plain `Up` (2026-10-18); the homepage card probes `127.0.0.1:8081`.
## Trusted Group (limiter.toml)
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on (`server.limiter: true`, `shared/searxng/settings.yml:11` — and the running instance confirms it: `GET /config` reports `limiter.enabled: true`, 2026-10-18). The file deploys to `/opt/searxng/config/limiter.toml` (`scripts/deploy-remote.sh:211`), mounted at `/etc/searxng/limiter.toml`. The deployed copy is byte-identical to the repo file today (`diff` clean, 2026-10-18).
Two lists control access:
- `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
- `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`) — `shared/searxng/limiter.toml:54-60`. An IP in `pass_ip` skips every bot-detection check, including the rate limiter: ip_lists "outranks every other method" (`limiter.toml:41-45`). `deploy.sh` copies the file into place on each deploy (`scripts/deploy-remote.sh:208-211`).
Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.
The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.
Reach SearXNG through Caddy (`search.franksdank.cloud`, guarded) for browser
use. A local subagent can hit the published port directly because its
use: the vhost challenges through `anubis:8923` (`dankclaw/Caddyfile:102-103`) and the
post-auth listener :8017 routes it to `192.168.0.143:8081` (`dankclaw/Caddyfile:361`).
A local subagent can hit the published port directly because its
slirp/loopback/LAN sources are in `pass_ip`. The searxng network
(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`, so Caddy's
`X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it
stays stable across reboots.
(`systemd-searxng`, 10.89.2.0/24) is in `trusted_proxies`
(`shared/searxng/limiter.toml:22-29`), so Caddy's `X-Forwarded-For` is honored. The
subnet is pinned in `searxng.network:16`, so it stays stable across reboots — that is
what keeps the `trusted_proxies` entry meaningful.
## Unblocking Google & Search Providers
The current request policy allows a modest amount of VPN latency (`4.0s` per
request, `12.0s` maximum) while limiting retries to one. This can recover slow
The current request policy allows a modest amount of VPN latency (`5.0s` per
request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
retries to one (`:32`). This can recover slow
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
disables observed CAPTCHA/429 providers from normal aggregation, and disables
pagination to reduce upstream request volume. SearXNG **suspends** offending
engines for hours to days, which is what shows up as "Google went unavailable".
This is upstream, not local.
VPN exit IP. SearXNG queries every engine it has enabled from one Gluetun VPN exit
IP: the running build ships 262 engines and **75 are enabled**
(`curl http://127.0.0.1:8081/config` → `engines[]`, 2026-10-18). The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
The default provider policy therefore favors anonymous Bing and DuckDuckGo (both `enabled: true`
in `/config`), disables observed CAPTCHA/429 providers from normal aggregation
(google, google cse, google news, google scholar, the whole `brave.*` family:
`enabled: false` on 2026-10-18), and disables pagination for those engines to reduce
upstream request volume. SearXNG **suspends** offending engines — which is what shows
up as "Google went unavailable". The default suspension is shorter than the folklore:
see the table in [[SearXNG Providers]]. This is upstream, not local.
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
@@ -165,10 +180,11 @@ root cause, the suspend-model table, the `engines:` config, and the
### Gotchas
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
- **Agent-API contract (verified live 2026-08-31):** the guarded path
(`search.franksdank.cloud`) answers non-browser API clients with Anubis'
- **Agent-API contract (verified live 2026-08-31, re-verified 2026-10-18):** the guarded
path (`search.franksdank.cloud`) answers non-browser API clients with Anubis'
JS-challenge page at HTTP **200** — a status-code-only client silently gets
HTML where it expected JSON. Check `content-type`, not just status. There is
HTML where it expected JSON (`curl` with a JSON query through the vhost returned
`200 text/html`, 2026-10-18). Check `content-type`, not just status. There is
deliberately no way for an API client through the guard (JS challenge + UA
deny-list); the agent path is the direct published port
`http://192.168.0.143:8081/search?q=…&format=json` from trusted sources —
@@ -176,38 +192,95 @@ root cause, the suspend-model table, the `engines:` config, and the
LAN by construction (no public route to the LAN IP). No second agent-only
instance is needed; a duplicate would double upstream request volume from the
same rate-sensitive VPN exit IP for zero benefit.
- The config file is root-owned (uid of the rootless podman namespace). The
official path is `deploy.sh --service searxng`: it copies the repo settings
- The config dir is owned by uid 100976 — the searxng container's mapped root inside
the rootless user namespace, not a host root account
(`ls -la /opt/searxng/config` 2026-10-18: both files `100976 100976`).
The official path is `deploy.sh --service searxng`: it copies the repo settings
and renders the `${VAR}` placeholders via `scripts/searxng-render-settings.py`
from `/opt/searxng/.env`. A hotfix can still hand-copy with `sudo cp`, but
remember the renderer: an unrendered copy silently serves the literal
from `/opt/searxng/.env` (`scripts/deploy-remote.sh:208-216`). A hotfix can still hand-copy with `sudo cp`, but remember the renderer: an unrendered copy silently serves the literal
placeholders (before 2026-09-26 the live `secret_key` literally was
`"${SEARXNG_SECRET}"` and nobody noticed).
- `base_url` must match the public address or proxy rewrites break.
- Keep the valkey cache container on the same podman network as searxng.
`"${SEARXNG_SECRET}"` and nobody noticed). Today's deployed file carries no
unrendered placeholder — the only `${` left in it is inside the header comment
(`grep -n '[$]{' /opt/searxng/config/settings.yml`, 2026-10-18).
- `base_url` must match the public address or proxy rewrites break — **the deployed
value does not**: `SEARXNG_BASE_URL` in `/opt/searxng/.env` is set so that the
rendered config reads `base_url: "http://192.168.0.143:8081/"`
(`/opt/searxng/config/settings.yml:10`, 2026-10-18), i.e. the published port on the
LAN IP, not `https://search.franksdank.cloud`. Browser use through the vhost still
works because Caddy rewrites `Host`, but absolute links SearXNG builds point at the
LAN address. The unit also passes `SEARXNG_BASE_URL` into the container
(`searxng.container:21`), and that is one of the variables SearXNG substitutes at
load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
config change, out of scope for docs.
- Keep the valkey cache container on the network SearXNG can actually reach — see
[[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
netns), so the network that matters is the one Gluetun and `searxng-valkey` share.
## Kagi (paid API engine)
Added 2026-09-26 (user-ordered). The stock SearXNG image ships a native
`kagi` engine module (`searx/engines/kagi.py`); it authenticates with
`Authorization: Bearer <key>` against `POST https://kagi.com/api/v1/search`.
Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
and declares `require_api_key: True` (`kagi.py:63`).
- **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
`[general, web]`, shortcut `kg`, `kagi_categ: search`.
`[general, web]`, shortcut `kg`, `kagi_categ: search`
(`shared/searxng/settings.yml:66-77`); live `/config` reports `kagi` with
`enabled: true`, categories `general, web` (2026-10-18).
- **Cost control**: `paging: false` — exactly one API credit per aggregated
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0).
query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
(`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
`kagi.py:67`, so the `false` here is a real override).
- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
`deploy-remote.sh` bakes the real key into the live settings.yml via
`scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
(dank-owned, mode 600). The key never enters git and never appears in any
process argv. SearXNG itself does NOT expand env vars in settings.yml —
without the renderer the engine fails `setup()` and is dropped at load.
(owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
The key never enters git and never appears in any
process argv. SearXNG does not expand *arbitrary* `${VAR}` placeholders in
settings.yml — it substitutes only a fixed set of documented environment variables
(`SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `SEARXNG_VALKEY_URL`, …; see the header
comments in upstream `searx/settings.yml` at the pinned commit), and `${KAGI_API_KEY}`
is not one of them. Without the renderer the engine fails `setup()` and is dropped
at load (`kagi.py:91` returns the error when the key is empty). The deployed file
has no unrendered placeholder left (2026-10-18).
- **Immune to the VPN-exit-IP problem**: Kagi authenticates, so it does not
care about the shared Gluetun exit IP reputation (see [[SearXNG Providers]]).
- **Verified live 2026-09-26**: direct API probe through the gluetun netns
(200, `data.search[]` shape), then a live aggregated query returned
29/29 results tagged `kagi`, zero unresponsive engines.
29/29 results tagged `kagi`, zero unresponsive engines. Re-checked 2026-10-18 only
as far as the config goes (engine enabled, key rendered); no paid query was fired
for this audit.
## Live engine census (2026-10-18)
`GET http://127.0.0.1:8081/config` answers `engines[]` for the running instance
(200 today; the endpoint would be blocked on a public instance, and
`public_instance: false` here). Counted from that response:
- 262 engines in the build, **75 enabled**, categories of the enabled set led by
`other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
`github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
`arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
`wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
`youtube`, `vimeo`, `dailymotion`, `bing videos`.
- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
`brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
- The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
engines that **no longer exist in this build** — `/config` lists no `startpage`,
`startpage images` or `startpage news`, so those overrides are no-ops kept as
intent. *(labelled 2026-10-18)*
## Citations
- `dankclaw/searxng/quadlet/searxng.container`, `gluetun.container`, `searxng-valkey.container`, `searxng.network`
- `shared/searxng/settings.yml`, `shared/searxng/limiter.toml`
- `scripts/deploy-searxng-quadlet.sh:11`, `scripts/deploy-remote.sh:208-216,583-593`
- `dankclaw/Caddyfile:102-103,361` (guarded vhost → post-auth → `192.168.0.143:8081`)
- Host evidence 2026-10-18: `podman ps`, `podman inspect searxng`, `podman network inspect systemd-searxng`, `ss -ltn`, `curl 127.0.0.1:8081/` → 200, `/healthz` → 200, `/config` → 200 (engine census), `/search?format=json` → 200 `application/json`, `diff shared/searxng/limiter.toml /opt/searxng/config/limiter.toml` (identical), `systemctl --user is-enabled/is-active gluetun searxng searxng-valkey`
## See Also