diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 0f71a97..1d4fc86 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -46,12 +46,20 @@ The honest summary: metrics and logs are collected and retained, one dashboard v ## Deployment -All five are rootless **Quadlet units** on Dankclaw. The units live in `dankclaw/observability/quadlet/.container` and install to `~/.config/containers/systemd/`, generating `.service`. +Six units now, across two hosts. Dankclaw runs five (Prometheus, Grafana, Loki, +Alloy, node-exporter) from `dankclaw/observability/quadlet/`; **Dankserver runs +its own node-exporter** from `dankserver/observability/quadlet/`, added +2026-09-16 after the same reboot argument took it down a second time — it had +been a compose-only service, died `Exited (143)` at a reboot, and left +Prometheus' `node-dankserver` job scraping nothing for a day and a half (its +compose form also bound the wildcard address). The deployer installs from +`"$TARGET/observability/quadlet"`, so each host owns its own units. -They left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots. +The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.** ```bash -./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter +./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter +./scripts/deploy.sh dankserver --service node-exporter ``` `deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first. @@ -63,7 +71,7 @@ Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podm The rule is the same as for every other service: the only off-host path is a guarded vhost (`.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong: - **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain. -- **node-exporter is the one deliberate LAN listener.** Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: `127.0.0.1:9100` for the homepage status probe and `192.168.0.143:9100` for Prometheus. It authenticates nothing, so it never gets a vhost. +- **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost. ## Scrape surface