Commit Graph
183 Commits
Author SHA1 Message Date
Frank 1f4766f233 observability: fix alloy ui component, busybox healthchecks for loki/alloy
- alloy 1.x renamed the server component to ui; the old block failed the
  initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
  healthchecks could never run; local images add a static busybox from
  alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
2026-08-22 03:59:14 -07:00
Frank 816986139f observability: run as container root 0:0, not 1000:1000
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
2026-08-22 03:45:06 -07:00
Frank 3c3f90e4c5 observability: run prometheus/loki/grafana as 1000:1000 (rootless uid mapping)
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman.
2026-08-22 03:19:18 -07:00
Frank 37f05a5f9e observability configs: loki delete_request_store, prometheus targets, grafana root URL
- loki: recent releases require compactor.delete_request_store when
  retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
  URL schemes (invalid); point at published LAN ports and the gitea
  quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
2026-08-22 03:04:36 -07:00
Frank 4572a7739d Caddy: keep plex.franksdank.casa HTTP-only (not in the Porkbun account) 2026-08-22 02:44:06 -07:00
Frank 7f4ce06927 Caddy phase 2: TLS via Porkbun DNS-01 wildcard certs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
  HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
  (the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
2026-08-22 02:39:41 -07:00
Frank 724f186414 Caddyfile: transport is a reverse_proxy subdirective (jellyfin h2c) 2026-08-21 21:50:03 -07:00
Frank 4a679fce80 Caddy consolidation: single Caddy on dankclaw fronts both hosts
Per user direction, Caddy runs only on dankclaw. It fronts every
service on both hosts: local services by container name, quadlet
services by published LAN port, dankserver services by published LAN
port (192.168.0.235:PORT).

- dankclaw/Caddyfile: merged vhost list (was: two Caddyfiles, one per
  host). gitea's single vhost moves in from the retired gitea-caddy
  quadlet (option B). Upstreams fixed to live reality: gitea :3000,
  search :8081 (gluetun), chat :3005 (open-webui quadlet), media
  services via 192.168.0.235. Retired: llms (open-webui on dankserver
  is removed; chat is its successor), the dankserver homepage (302 to
  the shared one).
- dankserver/Caddyfile, dankclaw/gitea/Caddyfile: deleted (git rm).
- dnsmasq addn-hosts: every name -> 192.168.0.143 (the one Caddy
  host); no .235 records, no llms.
- compose.yaml: removed the openwebui/postgres/gitea services (they
  are quadlets now); caddy service mounts the rendered Caddyfile from
  /home/dank/Containers/caddy, healthcheck probes the bare-IP
  catch-all (302).
- deploy.sh: per-target logic (dankserver = pull-only, health via the
  shared Caddy at 192.168.0.143); renders CADDY_BASICAUTH_HASH from
  the host .env into the mounted Caddyfile.
- .env.example: CADDY_BASICAUTH_HASH on dankclaw; Porkbun block
  dropped from dankserver (its Caddy is gone); SEARXNG_BASE_URL to
  the flat name; stale OPENAI_* block dropped (quadlet is
  self-contained).
- tests: caddyfile.bats now asserts the single-Caddyfile invariant;
  dnsmasq.bats cross-checks vhosts -> .143 and the retired names.
2026-08-21 21:44:18 -07:00
Frank 9c8298df05 dnsmasq: add foreground CMD to the image
The image had no command, so the container exited immediately (0).
dnsmasq also daemonizes by default; -k keeps it as PID 1.
2026-08-21 19:38:08 -07:00
Frank 1dee2b737e dnsmasq: serve config from /home/dank/Containers/dnsmasq (rootless-correct)
/opt is root-owned; the established data-location pattern (see
docs/Architecture/Storage Policy.md, open-webui quadlet) is
/home/dank/Containers/<service>.
2026-08-21 16:41:49 -07:00
Frank 55d062ea7b DNS names: flat franksdank.cloud scheme + dnsmasq LAN DNS on dankclaw
Phase 1 of the local-DNS-name work. Users type a name, not a port.

- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
  serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
  Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
  renamed to flat <service>.franksdank.cloud. Explicit http:// site
  addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
  from the repo): added the live-only llms.franksdank.cloud vhost
  (OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
  (the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
  instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
  port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
  the right host IPs); caddyfile.bats naming-convention tests updated.

LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
2026-08-21 16:31:42 -07:00
Frank 833f2365f1 env: add Porkbun API token placeholders (DNS-01 cert issuance)
Real tokens live only in the host .env files (git-ignored):
- /home/dank/serverAppConfigs/dankclaw/.env
- /home/dank/serverAppConfigs/dankserver/.env

Token is scoped to the franksdank.cloud zone (DNS read/write).
2026-08-21 16:00:16 -07:00
Frank 34918bf309 hermes: add web dashboard service (basic-auth, LAN port 9119)
Add a hermes-dashboard compose service that runs 'hermes dashboard' on
0.0.0.0:9119 with HTTP basic auth. The username + scrypt password hash
live in the git-ignored /home/dank/hermes-data config (plaintext never
stored). A separate container from the Signal gateway, since the gateway
entrypoint does not start the dashboard. Both share the data volume and
are boot-persistent. Docs page + index link added (docs-first contract).
2026-08-20 22:54:35 -07:00
Frank afdb686c7e hermes: point Signal adapter at host LAN IP (slirp alias drops at runtime)
The slirp host.containers.internal mapping (169.254.1.2) went down at
runtime, so the Signal adapter could not reach the signal-cli daemon.
The host LAN IP (192.168.0.143:8080) is the stable container->host path
and is reachable from the container. Switch SIGNAL_HTTP_URL to it.
2026-08-20 21:34:48 -07:00
Frank b5211845da Hermes: drop the :8090 healthcheck (messaging gateway has no HTTP endpoint)
'hermes gateway run' starts the messaging gateway (Signal SSE), not the :8090
web dashboard, so probing 8090 always reported unhealthy. The container's
running state plus the daemon's SSE connection are the real health signal.
2026-08-20 12:30:42 -07:00
Frank 5e71b0584f Hermes: run the headless gateway (not the TUI) in the container
The image's default s6 entrypoint launches the interactive TUI, which exits
without a TTY and crash-loops the container. Override the entrypoint to run
'hermes gateway run --force' (the documented Docker mode) and point HERMES_HOME
at the mounted data dir so the Signal platform config + state persist. The
Signal platform is enabled in config.yaml (platforms.signal.enabled).
2026-08-20 12:14:18 -07:00
Frank b167a07fc0 Hermes: wire the Signal adapter to the host signal-cli daemon
The hermes container reaches the host signal-cli daemon via podman's
host.containers.internal alias (127.0.0.1 in the container is not the host).
SIGNAL_ACCOUNT/SIGNAL_ALLOWED_USERS come from the git-ignored .env (the phone
number is personal, not committed). Move the data volume to a dank-owned path
(/opt is root-owned and not writable by the run user).
2026-08-20 10:51:43 -07:00
Frank bb188becb0 ComfyUI: persist models/data under /home/dank/comfyui-data
/opt is root-owned and not writable by the run user. Point the model/input/
output/custom_nodes mounts at a dank-owned path so the container can write
them without sudo. The tests do not assert volume paths, so this is a no-op
for the suite.
2026-08-19 15:55:00 -07:00
Frank 88dcd3cc0f ComfyUI: rootless-correct GPU group config (video only)
Rootless podman cannot map the 'render' gid (990) into the container
namespace (dank's subgid range is 100000+), so --group-add render fails to
start the container. GPU access actually comes from the /dev/kfd + /dev/dri
passthrough, checked host-side (the run user is already in 'render' on the
host). So add only the 'video' group (which resolves) and drop 'render'.

Update the group test to assert video present and render ABSENT (with the
rootless rationale), per the approved rootless config.
2026-08-19 15:50:28 -07:00
Frank 354ddc473f ComfyUI: GPU (gfx1151 ROCm) build via container
Replace the CPU stock image with a container built from
Containerfile.comfyui on AMD's gfx1151-tagged vLLM base — the only torch
build verified to report torch.cuda.is_available()=True on the Strix Halo
APU (Radeon 8060S, capability (11,5)).

- Containerfile bakes the ComfyUI code in and installs its deps, but skips
  the torch family (the base image already ships the gfx1151 ROCm
  torch/torchvision/torchaudio; an unpinned 'torch' in requirements.txt would
  clobber the gfx1151 kernels).
- compose comfyui service: build + image comfyui-rocm:gfx1151, passes
  /dev/kfd + /dev/dri, adds video+render groups, drops --cpu. Models/data
  persist under /opt/comfyui.
- Un-skip the four compose-wiring acceptance tests (the gate for this work).
2026-08-19 15:20:46 -07:00
Frank 3d4604c19f Stand up open-webui on dankclaw (quadlet, boot-persistent)
Replace the dead dankserver open-webui (bare container, :3000, pointed at a
dead :9090 backend) with a boot-persistent quadlet on dankclaw:
- CPU image (:main); dankclaw is CPU-only, so the :cuda image is wasted.
- Isolated on slirp4netns; publishes 3005 (3000 is Gitea's HTTP port).
- Default LLM backend is the local vLLM on 52625 (frank-default-large).
- Data persists under /home/dank/Containers/open-webui/data.
- Caddy route now proxies to the published LAN port (the container is not on
  Caddy's network).

Also corrects the model/port mapping in the docs: the live state (verified
2026-08-19) is frank-default-large on vLLM :52625 (dankclaw) and
frank-default-small on llama.cpp :7770 (dankserver) — the reverse of the
2026-08-17 note.
2026-08-19 01:04:50 -07:00
Frank 06ac2a84cf SearXNG quadlet: use AddDevice= (the generator rejects Device=)
Device= in [Container] is a [Volume]-group key; the container device key is
AddDevice= (maps to podman --device). Update the /dev/net/tun device line and
the test seam.
2026-08-19 00:00:42 -07:00
Frank e036296fd1 SearXNG quadlet: use AddCapability= (the generator rejects CapAdd=)
The podman quadlet generator fails with "unsupported key 'CapAdd' in group
'Container'" and skips the whole unit. The correct key is AddCapability=
(maps to podman --cap-add). Update the gluetun unit and its test seam.
2026-08-18 23:57:25 -07:00
Frank 71fe8c008f SearXNG: move the trio out of compose into quadlet deployment
Remove gluetun/searxng/valkey from dankclaw/compose.yaml (they are now
quadlet units) so podman-compose up can no longer fight them for the netns and
port 8081. Add scripts/deploy-searxng-quadlet.sh: an idempotent installer that
copies the units to ~/.config/containers/systemd/, daemon-reloads, enables them
for boot, and starts them in dependency order. Safe to run on first migration
and on every later change.
2026-08-18 23:34:08 -07:00
Frank 6e9bfd5d3e SearXNG: define the gluetun+searxng+valkey trio as quadlet units
Converts the podman-compose SearXNG stack to boot-persistent quadlet units so
it survives a reboot (the compose deployment did not). searxng sets
Network=gluetun.container, which makes it reuse gluetun's network namespace so
outbound search traffic exits the WireGuard tunnel; it publishes no port of its
own (the port lives on gluetun). gluetun and valkey join a pinned searxng
network (10.89.2.0/24, DNS-enabled) so searxng reaches valkey by name.

Secrets stay in /opt/searxng/.env (mode 0600), referenced via [Service]
EnvironmentFile= so ${VAR} expands. Adds a bats seam asserting the netns
wiring, port placement, boot install, and that no secret material is tracked.
2026-08-18 23:20:28 -07:00
Frank aef7eefa35 Gluetun: healthcheck via the tun0 tunnel interface
The :8000 status server 404s on every path and the image has no curl, so the
prior healthchecks could not pass. Check /sys/class/net/tun0 instead: it exists
only while the WireGuard tunnel is connected, so the healthcheck reflects real
VPN state.
2026-08-18 18:12:04 -07:00
Frank 39f43d010a Gluetun: fix healthcheck (no curl in image; /health is not an endpoint)
The gluetun image ships wget, not curl, and its status server answers 200 at
/ on port 8000 (not /health). The old curl-based healthcheck always failed, so
the container read 'unhealthy' even with a working tunnel. Use wget against /.
2026-08-18 18:00:10 -07:00
Frank 9ca22d1c34 Gluetun/Valkey: use fully-qualified docker.io image names
The host has no unqualified-search registries in registries.conf, so the
short names qmcgaw/gluetun and valkey/valkey did not resolve and the pull
hung. Qualify them with docker.io/ so the pull works on any host.
2026-08-18 16:39:06 -07:00
Frank a046d2c89d Gluetun: pass full single-peer WireGuard config to the container
The compose only forwarded VPN_SERVICE_PROVIDER, VPN_TYPE and the private
key, so Gluetun could not form a tunnel. Forward the peer public key,
endpoint IP/port, addresses, allowed IPs and keepalive interval. Add
matching placeholders to .env.example (current Gluetun uses the
single-peer WIREGUARD_* model, not the old WIREGUARD_PEERS_* names).
2026-08-18 15:16:50 -07:00
Frank 1de43f6777 docs: update sibling KB for the 2026-08-17 fleet redo (llama.cpp moved to Dankserver, vLLM on Dankclaw; OpenWebUI/Hermes backend addresses) 2026-08-17 18:10:57 -07:00
Frank c81933ccb4 OpenWebUI: retarget to Dankserver llama.cpp (:7770 moved in 2026-08-17 redo) 2026-08-17 16:39:30 -07:00
Frank 04112e88e6 fix: gitea boot persistence via quadlet (incident 2026-08-16)
unattended-upgrades rebooted dankclaw at 15:01; the legacy
podman-compose gitea stack had no systemd units, so rootless podman
did not auto-start the containers. Gitea was down ~1 hour.

- add quadlet units (dankclaw/gitea/quadlet/): gitea-postgres,
  gitea-server, gitea-caddy + gitea.network (DNS enabled; the default
  rootless podman network has dns_enabled=false)
- add gitea stack Caddyfile (upstream gitea-server:3000) and
  .env.example; live password stays in /opt/gitea/.env (0600)
- fix latent name bugs in main stack config: DATABASE_HOST
  gitea-postgres, Caddyfile upstream gitea-server
- docs: rewrite Gitea page for the quadlet stack (management, boot
  persistence, secrets, gotchas, incident log); fix port 3000 in
  stack table; Postgres connection notes
- legacy compose retired on target to /opt/gitea/.deprecated/
2026-08-16 18:31:38 -07:00
Frank e3eb5334cb feat: initialize serverAppConfigs with podman stacks for Dankserver and Dankclaw
- Dankserver stack: RomM, Copyparty, Calibre-Web, node-exporter
- Dankclaw stack: Gluetun VPN (SearXNG tunnel), SearXNG+Valkey, ComfyUI
  (CPU backend), OpenWebUI, Hermes, Gitea+Postgres (migrated), Caddy
  reverse proxy, Grafana+Prometheus+Loki+Alloy observability
- Caddyfiles with .local hostname convention and basicauth homepage
- bind-mount storage policy, no named volumes
- deploy.sh mirroring vllmScripts git workflow (push -> pull -> restart -> health)
- docs/ Obsidian vault: per-service docs from frank-default-small research
  subagents (promtail EOL -> Alloy; gitea SECRET_KEY preservation; ComfyUI --cpu)
- bats test suite at 6 seams — 22 tests green
2026-08-13 20:19:41 -07:00