Two real bugs found on the first production deploy:
- /admin/ 404'd: file_server looked for /srv/homepage/admin/ (a dir), but the
page is admin.html. Add '@admin rewrite * /admin.html' so the /admin/ path
serves admin.html.
- The pages used relative refs (style.css, app.js, services.json, status.json),
which from /admin/ resolved to /admin/... (404). Add <base href="/"> so all
refs resolve to the site root on both pages.
- status.json was written 0600 (root); the checker now chmods it 0644 so Caddy
can serve it regardless of its user.
Red tests first, then the pages.
- tests/homepage_pages.bats: the pages reference the data + theme; no
auto-refresh (Q41); no cursor alteration (Q28); the styling toggle strips
to bare HTML (Q27); the footer shows the last master deployment datetime
(Q43).
- index.html / admin.html: the user and admin pages. A grid of TUI panels,
a summary line, the bare toggle, the deployed-at footer.
- app.js: reads status.json + services.json once (no auto-refresh), renders
one panel per service for the page's category, the toggle, the footer.
- style.css: the terminal TUI theme (green-on-black, phosphor palette,
scanlines) + a ?bare=1 mode that strips to unstyled HTML. No custom cursor.
Functionally verified with a Node DOM shim: 9 user panels, correct summary,
correct footer datetime.
Red tests first (the old services.json was an object, not the new list schema).
- tests/homepage_config.bats: the config is a JSON list; every entry has
name/category/probe; category is user|admin; method is tcp|http; a
showLink entry has a url and a status-only entry does not.
- shared/homepage/services.json: the real fleet. User: SearXNG, ComfyUI,
OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin:
Hermes, Grafana, Prometheus, Loki, node-exporter, alloy, Caddy, dnsmasq,
Gluetun (wip - a tunnel has no simple local port), the two LLMs, mariadb,
valkey, Postgres. Ports confirmed from the repo config (compose/quadlet).
docs fields are empty for now (the /docs/ workstream fills them as each
service goes active).
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
Phase 1 of the local-DNS-name work. Users type a name, not a port.
- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
renamed to flat <service>.franksdank.cloud. Explicit http:// site
addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
from the repo): added the live-only llms.franksdank.cloud vhost
(OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
(the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
the right host IPs); caddyfile.bats naming-convention tests updated.
LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
Root cause of the 429s: the ip_limit rate limiter caps JSON (format!=html)
requests at API_MAX=4 per IP per hour, and that constant is hardcoded in
Python (searx/botdetection/ip_limit.py), not settable in limiter.toml. The
only bypass is pass_ip (filter_request returns before ip_limit runs).
Rootless podman strips the client IP on published-port hits, so the container
sees a slirp/bridge source (10.88.x/10.89.x), never the real client - hence
even a LAN-IP curl 429'd on /search. Expand pass_ip to loopback + the host's
private podman/slirp ranges (plus the existing LAN) so local subagents present
a trusted source and skip the rate limit entirely. All these ranges are
reachable only from the host or LAN, never the internet.
Converts the podman-compose SearXNG stack to boot-persistent quadlet units so
it survives a reboot (the compose deployment did not). searxng sets
Network=gluetun.container, which makes it reuse gluetun's network namespace so
outbound search traffic exits the WireGuard tunnel; it publishes no port of its
own (the port lives on gluetun). gluetun and valkey join a pinned searxng
network (10.89.2.0/24, DNS-enabled) so searxng reaches valkey by name.
Secrets stay in /opt/searxng/.env (mode 0600), referenced via [Service]
EnvironmentFile= so ${VAR} expands. Adds a bats seam asserting the netns
wiring, port placement, boot install, and that no secret material is tracked.
A direct hit on the published port has no X-Forwarded-For header, so the
bot-detection cannot resolve a client and returns 429. SearXNG must be
reached through Caddy (search.dankclaw.local), which sets X-Forwarded-For.
Add the podman bridge subnets to trusted_proxies so Caddy's source is
trusted for XFF extraction; note the CIDR is dynamic and should be pinned.
Verified agentic JSON search returns 26-31 relevant results via this path.
The trusted group lives in limiter.toml, not settings.yml. Add
shared/searxng/limiter.toml trusting 192.168.0.0/24 in pass_ip and
trusted_proxies, copy it into /opt/searxng/config on deploy, and correct
the SearXNG doc. New test seam: tests/searxng_limiter.bats.
Frank