Files

T

Frank 701ab14f60 docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.

2026-09-11 16:10:27 -07:00

2.5 KiB

Raw Blame History

Node-Exporter

Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.

Hosts: Dankclaw (unit) + Dankserver (compose service) — Docs: https://prometheus.io/docs/guides/node-exporter

Facts

Item	Value
Image	`prom/node-exporter:latest`
Listen addresses	`127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw)
Network	`Network=host` + `--pid=host`
Unit	`dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service`
Paths	`--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs`
Auth	none — it is an unauthenticated metrics endpoint by design

Why it is the one LAN listener

Every other observability service publishes on loopback. This one cannot:

Prometheus runs in a container on the dankclaw_default network and cannot reach the host's loopback address, so it needs the host's LAN IP; shared/prometheus/prometheus.yml scrapes 192.168.0.143:9100 under the node-dankclaw job.
The homepage status timer runs on the host and probes 127.0.0.1:9100.

The unit therefore binds exactly those two addresses with two --web.listen-address flags — never the wildcard. It has no authentication, so it is never vhosted and never published beyond the LAN.

Mounts

Container path	Host path
`/host/proc`	`/proc` (read-only)
`/host/sys`	`/sys` (read-only)
`/rootfs`	`/` (read-only)

The mount paths and the --path.* flags are a pair: changing one without the other gives an exporter that starts cleanly and reports nothing.

Prometheus Targets

Host	Target
Dankclaw	`192.168.0.143:9100`
Dankserver	`192.168.0.235:9100`

The scrape jobs live in shared/prometheus/prometheus.yml; see Prometheus. The tests/observability_quadlet.bats seam pins the Dankclaw address to this unit's listen address so the two cannot drift apart.

Gotchas

Host networking is required (or a published port); the deploy's health wait probes 127.0.0.1:9100/metrics.
The /proc, /sys and root mounts must be read-only.
Each host's instance reflects only its own host; do not try to share one.
Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics.

2.5 KiB Raw Blame History