dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: six observability units across two hosts, and what the reboot cost

Browse Source 
The page said "all five are rootless Quadlet units on Dankclaw", which stopped
being true when Dankserver's node-exporter was restored to a unit. It now names
both hosts and both deployments, records why the second one happened (the same
compose-only reboot argument, a day and a half of missing host metrics, and a
wildcard bind), and states the exposure rule for a two-listener fleet with the
`ss` evidence that the wildcard is gone.
This commit is contained in:
Frank
2026-09-16 07:09:55 -07:00
parent da9cff046e
commit d8835e69c6
1 changed files with 12 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Observability.md
+12 -4
View File
@@ -46,12 +46,20 @@ The honest summary: metrics and logs are collected and retained, one dashboard v
## Deployment
All five are rootless **Quadlet units** on Dankclaw. The units live in `dankclaw/observability/quadlet/<name>.container` and install to `~/.config/containers/systemd/`, generating `<name>.service`.
Six units now, across two hosts. Dankclaw runs five (Prometheus, Grafana, Loki,
Alloy, node-exporter) from `dankclaw/observability/quadlet/`; **Dankserver runs
its own node-exporter** from `dankserver/observability/quadlet/`, added
2026-09-16 after the same reboot argument took it down a second time — it had
been a compose-only service, died `Exited (143)` at a reboot, and left
Prometheus' `node-dankserver` job scraping nothing for a day and a half (its
compose form also bound the wildcard address). The deployer installs from
`"$TARGET/observability/quadlet"`, so each host owns its own units.
They left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots.
The Dankclaw five left `dankclaw/compose.yaml` on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. **What runs is now the unit; the compose file is the historical record.**
```bash
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter
./scripts/deploy.sh dankclaw --service loki # or prometheus|grafana|alloy|node-exporter
./scripts/deploy.sh dankserver --service node-exporter
```
`deploy_observability` in `scripts/deploy-remote.sh` installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and `SERVICE_ORDER` in the planner puts each dependency first.
@@ -63,7 +71,7 @@ Every unit carries a healthcheck, so a wedged process reads `unhealthy` in `podm
The rule is the same as for every other service: the only off-host path is a guarded vhost (`<name>.franksdank.cloud` → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:
- **Prometheus, Grafana, Loki and Alloy publish on `127.0.0.1` only.** Caddy reaches them by container name over the `dankclaw_default` network, and the host-side homepage status timer probes loopback. Publishing on `0.0.0.0` — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
- **node-exporter is the one deliberate LAN listener.** Prometheus runs in a container and cannot reach the host's loopback, so the unit binds exactly two addresses: `127.0.0.1:9100` for the homepage status probe and `192.168.0.143:9100` for Prometheus. It authenticates nothing, so it never gets a vhost.
- **node-exporter is the one deliberate LAN listener, and there are two of them now — one per host.** Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (`127.0.0.1:9100`) for host-local probes and its own LAN address (`192.168.0.143:9100` on Dankclaw, `192.168.0.235:9100` on Dankserver) for Prometheus. `ss -ltn` on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.
## Scrape surface