Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

4.6 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Node-Exporter

Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.

Hosts: Dankclaw (unit) + Dankserver (unit) — Docs: https://prometheus.io/docs/guides/node-exporter

Both hosts run the same shape as a Quadlet unit: dankclaw/observability/quadlet/node-exporter.container and dankserver/observability/quadlet/node-exporter.container (the compose-only Dankserver copy died for a day and a half before anyone noticed, so it became a unit on 2026-09-16). Both report generated + active on the host and bind their own two addresses (systemctl --user is-enabled|is-active node-exporter, ss -ltnp on dankserver, 2026-10-18); tests/observability_quadlet.bats:419 pins the Dankserver side to a unit, not compose.

Facts

Item	Value
Image	`prom/node-exporter:latest`
Listen addresses	`127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw, `ss -ltnp` 2026-10-18; Dankserver binds `127.0.0.1:9100` + `192.168.0.235:9100`)
Network	`Network=host` + `--pid=host` (`dankclaw/observability/quadlet/node-exporter.container:37-38`)
Unit	`dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service`
Paths	`--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` (`node-exporter.container:39`)
Auth	none — it is an unauthenticated metrics endpoint by design

Why it is the one LAN listener

Every other observability service publishes a port on loopback. This one runs in the host network namespace instead, and netavark DNATs only published ports — so a bridge container cannot reach a host-netns bind at the host's LAN address, and the exporter has to be reachable at an address the consumers can actually dial:

Prometheus runs in a container on dankclaw_default and scrapes host.containers.internal:9100 (shared/prometheus/prometheus.yml:37-39) — the gateway address netavark delivers a bridge container's host-bound dial to. It does not scrape 192.168.0.143:9100; that address is refused by construction from this container, the same rule that sends frank-dragoon and frank-zealot to the gateway. node-dankserver is a different machine, so its LAN address is used (prometheus.yml:41-43).
The homepage status timer runs on the host and probes 127.0.0.1:9100 (shared/homepage/services.json:220).

The unit therefore binds exactly those two addresses with two --web.listen-address flags (node-exporter.container:39) — never the wildcard: ss -ltnp (2026-10-18) shows 127.0.0.1:9100 and 192.168.0.143:9100 and nothing else. The LAN bind stays in the unit because LAN machines scrape it directly and the seam test pins it, not because this Prometheus needs it. It has no authentication, so it is never vhosted and never published beyond the LAN.

Mounts

Container path	Host path
`/host/proc`	`/proc` (read-only)
`/host/sys`	`/sys` (read-only)
`/rootfs`	`/` (read-only)

The mount paths and the --path.* flags are a pair: changing one without the other gives an exporter that starts cleanly and reports nothing.

Prometheus Targets

Host	Target
Dankclaw	`host.containers.internal:9100` (same-host bridge container → the gateway, `prometheus.yml:39`; live target `http://host.containers.internal:9100/metrics` is up, `curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18)
Dankserver	`192.168.0.235:9100`

The scrape jobs live in shared/prometheus/prometheus.yml; see Prometheus. The tests/observability_quadlet.bats seam pins both halves: the unit's two --web.listen-address values (:103-104) and the node-dankclaw target string "host.containers.internal:9100" (:108-131), so the exporter and its only Prometheus consumer cannot drift apart.

Gotchas

Host networking is required (or a published port); the deploy's health wait probes 127.0.0.1:9100/metrics (scripts/deploy-remote.sh:536) and the unit's HealthCmd the same address from inside the host namespace (node-exporter.container:43).
The /proc, /sys and root mounts must be read-only.
Each host's instance reflects only its own host; do not try to share one.
Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics.

4.6 KiB Raw Blame History