dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Browse Source 
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.
This commit is contained in:
Frank committed 2026-09-11 16:10:27 -07:00
1 parent 25c93984ad
commit 701ab14f60
8 files changed
+220 -147
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Infrastructure/Node-Exporter.md
+32 -31
View File
@@ -1,55 +1,56 @@
# Node-Exporter
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host inside its own container.
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.
**Hosts:** Dankserver + Dankclaw — **Docs:** https://prometheus.io/docs/guides/node-exporter
**Hosts:** Dankclaw (unit) + Dankserver (compose service) — **Docs:** https://prometheus.io/docs/guides/node-exporter
## Facts (from research)
## Facts
| Item | Value |
|------|-------|
| Image | `prom/node-exporter:latest` |
| Port | 9100 (host) |
| Network | `host` mode |
| Config | `--path.procfs=/host/proc`, `--path.sysfs=/host/sys`, `--path.rootfs=/rootfs` |
| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw) |
| Network | `Network=host` + `--pid=host` |
| Unit | `dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service` |
| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` |
| Auth | none — it is an unauthenticated metrics endpoint by design |
## Configuration
## Why it is the one LAN listener
```yaml
services:
node-exporter:
image: prom/node-exporter:latest
restart: always
network_mode: host
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/rootfs
```
Every other observability service publishes on loopback. This one cannot:
- **Prometheus** runs in a container on the `dankclaw_default` network and cannot reach the host's loopback address, so it needs the host's LAN IP; `shared/prometheus/prometheus.yml` scrapes `192.168.0.143:9100` under the `node-dankclaw` job.
- **The homepage status timer** runs on the host and probes `127.0.0.1:9100`.
The unit therefore binds exactly those two addresses with two `--web.listen-address` flags — never the wildcard. It has no authentication, so it is never vhosted and never published beyond the LAN.
## Mounts
| Container path | Host path |
|----------------|-----------|
| `/host/proc` | `/proc` (read-only) |
| `/host/sys` | `/sys` (read-only) |
| `/rootfs` | `/` (read-only) |
The mount paths and the `--path.*` flags are a pair: changing one without the other gives an exporter that starts cleanly and reports nothing.
## Prometheus Targets
Prometheus scrapes each host's node-exporter directly:
| Host | Target |
|------|--------|
| Dankserver | `192.168.0.235:9100` |
| Dankclaw | `192.168.0.143:9100` |
| Dankserver | `192.168.0.235:9100` |
These scrape jobs live in the Prometheus config; see [[Prometheus]] and [[Observability]].
The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins the Dankclaw address to this unit's listen address so the two cannot drift apart.
## Gotchas
- Requires `network_mode: host` (or a published port) so Prometheus can reach 9100.
- The `/proc`, `/sys`, and root mounts must be `read-only` (`:ro`).
- Do not mount the same paths on both hosts' containers; each node-exporter reflects only its own host.
- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics`.
- The `/proc`, `/sys` and root mounts must be read-only.
- Each host's instance reflects only its own host; do not try to share one.
- Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics.
## See Also
- [[Prometheus]] — the scraper
- [[Observability]] — the full stack
- [[Observability]] — the full stack, including what is still unconfigured