The LLM fleet documentation (deployment runbook, model runbooks,
benchmarks, research archive, retired-topology history) joins the
project knowledge base as docs/LLM/. Wikilinks resolve by filename,
so no link updates are needed for the move.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
Add a hermes-dashboard compose service that runs 'hermes dashboard' on
0.0.0.0:9119 with HTTP basic auth. The username + scrypt password hash
live in the git-ignored /home/dank/hermes-data config (plaintext never
stored). A separate container from the Signal gateway, since the gateway
entrypoint does not start the dashboard. Both share the data volume and
are boot-persistent. Docs page + index link added (docs-first contract).
ComfyUI-Manager (V3.41) is installed in the mounted custom_nodes dir and
self-installs its pip deps on start. Document the install command, the
bind-mount-shadows-image gotcha, and the /manager/version health check.
Rewrite ComfyUI.md to match reality: comfyui-rocm:gfx1151 built from
Containerfile.comfyui on AMD's gfx1151 base, device passthrough, video group
only (render is unmappable in rootless), data under /home/dank/comfyui-data,
boot-persistent via a user systemd unit. Mark research 'way 1' as applied and
update the repo-consequence section (the CPU config is no longer the default).
The JSON rate cap (API_MAX=4/hour) is hardcoded in Python, not settable in
limiter.toml; pass_ip is the only bypass. Rootless podman strips the client IP
on published-port hits (the container sees a slirp source), so local subagents
must present a pass_ip source. Document the slirp IP-strip, the pass_ip
expansion, and the root-owned config file (one-off sudo cp to update).
Replace the dead dankserver open-webui (bare container, :3000, pointed at a
dead :9090 backend) with a boot-persistent quadlet on dankclaw:
- CPU image (:main); dankclaw is CPU-only, so the :cuda image is wasted.
- Isolated on slirp4netns; publishes 3005 (3000 is Gitea's HTTP port).
- Default LLM backend is the local vLLM on 52625 (frank-default-large).
- Data persists under /home/dank/Containers/open-webui/data.
- Caddy route now proxies to the published LAN port (the container is not on
Caddy's network).
Also corrects the model/port mapping in the docs: the live state (verified
2026-08-19) is frank-default-large on vLLM :52625 (dankclaw) and
frank-default-small on llama.cpp :7770 (dankserver) — the reverse of the
2026-08-17 note.
Update SearXNG, Gluetun, and Valkey to describe the quadlet units instead of
compose: searxng sets Network=gluetun.container, the units install to
~/.config/containers/systemd/, and scripts/deploy-searxng-quadlet.sh deploys
them. Add a Boot Persistence section to the Stack doc. Correct the Gluetun
tunnel check (the :8000 server 404s; verify tun0 instead) and note the pinned
10.89.2.0/24 subnet.
A direct hit on the published port has no X-Forwarded-For header, so the
bot-detection cannot resolve a client and returns 429. SearXNG must be
reached through Caddy (search.dankclaw.local), which sets X-Forwarded-For.
Add the podman bridge subnets to trusted_proxies so Caddy's source is
trusted for XFF extraction; note the CIDR is dynamic and should be pinned.
Verified agentic JSON search returns 26-31 relevant results via this path.
The trusted group lives in limiter.toml, not settings.yml. Add
shared/searxng/limiter.toml trusting 192.168.0.0/24 in pass_ip and
trusted_proxies, copy it into /opt/searxng/config on deploy, and correct
the SearXNG doc. New test seam: tests/searxng_limiter.bats.
- README: fix broken 'Also See' placeholder URL -> live Gitea LAN URL
- index.md: expand vllmScripts See Also with the endpoints it serves
- OpenWebUI.md: :7770 backend is the llama.cpp PP head (split to the
3090), managed by vllmScripts; note the FLM NPU :52625 alternative;
fix the broken [[llama.cpp Server]] wikilink
- Gitea.md: hosted repositories table (both projects)
- Dankclaw Stack.md: host-level LLM endpoints in the service inventory
- AGENTS.md: runtime note — LLM endpoints live in vllmScripts, not here
Pre-existing failures (untracked ComfyUI WIP): tests 6, 8, 9, 10.
Docs seams green: ok 19 (compose service docs), ok 20 (index links).
unattended-upgrades rebooted dankclaw at 15:01; the legacy
podman-compose gitea stack had no systemd units, so rootless podman
did not auto-start the containers. Gitea was down ~1 hour.
- add quadlet units (dankclaw/gitea/quadlet/): gitea-postgres,
gitea-server, gitea-caddy + gitea.network (DNS enabled; the default
rootless podman network has dns_enabled=false)
- add gitea stack Caddyfile (upstream gitea-server:3000) and
.env.example; live password stays in /opt/gitea/.env (0600)
- fix latent name bugs in main stack config: DATABASE_HOST
gitea-postgres, Caddyfile upstream gitea-server
- docs: rewrite Gitea page for the quadlet stack (management, boot
persistence, secrets, gotchas, incident log); fix port 3000 in
stack table; Postgres connection notes
- legacy compose retired on target to /opt/gitea/.deprecated/
Live-verified on Dankclaw 2026-08-14:
- official rocm/comfyui image is built PYTORCH_ROCM_ARCH=gfx942;gfx950
(datacenter only) -> torch.cuda.is_available() False on gfx1151
- AMD gfx1151-tagged PyTorch build (rocm/vllm:...gfx1151...pytorch_2.9.1)
-> avail True, Radeon 8060S, cap (11,5); needs /dev/kfd + /dev/dri
- torch.backends.vulkan is compiled out of standard wheels; no ComfyUI
--vulkan flag exists; Vulkan is a dead end for ComfyUI
- Lemonade Server does image gen via wrapped stable-diffusion.cpp, whose
HIP kernels include gfx1151 -> works where ComfyUI's torch cannot
New pages: ComfyUI ROCm and Vulkan, Lemonade and sd.cpp; ComfyUI.md
qualified; compose stays --cpu until a gfx1151-enabled torch image is built
Frank