feat(observability): per-container metrics via cadvisor + Containers dashboard
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
This commit is contained in:
1 parent
961713e92d
commit
31908c623f
8 files changed
+398
-15
No files matched your search
@@ -131,3 +131,63 @@ scrape_configs:
|
||||
- targets: ["192.168.0.235:7770"]
|
||||
labels:
|
||||
service: frank-archon
|
||||
|
||||
# Per-container metrics from cadvisor (a rootless quadlet on each host).
|
||||
# cadvisor keys every series by the cgroup path; for a quadlet container that
|
||||
# path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
|
||||
# lives in it. The relabel below pulls that name out into a `container` label
|
||||
# and keeps only the per-unit aggregate — cadvisor's raw host root,
|
||||
# system.slice daemons and machine.slice scopes carry no such name and are
|
||||
# dropped, as is the per-CPU-core split (only cpu="total" survives), so the
|
||||
# scrape stays small. cadvisor cannot bind per-interface, so it is reached at
|
||||
# the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
|
||||
# is why node-dankclaw scrapes via host.containers.internal), a published port
|
||||
# IS reachable from this bridge container at the host LAN address.
|
||||
- job_name: "cadvisor-dankclaw"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.143:8080"]
|
||||
labels:
|
||||
host: dankclaw
|
||||
metric_relabel_configs:
|
||||
- source_labels: [id]
|
||||
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
|
||||
target_label: container
|
||||
replacement: '$1'
|
||||
# Keep only the unit-level cgroup; drop its payload/control children so a
|
||||
# container is not counted twice.
|
||||
- source_labels: [id]
|
||||
regex: '.*/\.service/.*'
|
||||
action: drop
|
||||
# Drop everything that is not a named unit (no `container` was extracted).
|
||||
- source_labels: [container]
|
||||
regex: '.+'
|
||||
action: keep
|
||||
# cpu="total" for cpu series; metrics with no cpu label (memory, network,
|
||||
# filesystem) are kept.
|
||||
- source_labels: [cpu]
|
||||
regex: '^$|total'
|
||||
action: keep
|
||||
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
||||
action: labeldrop
|
||||
|
||||
- job_name: "cadvisor-dankserver"
|
||||
static_configs:
|
||||
- targets: ["192.168.0.235:8080"]
|
||||
labels:
|
||||
host: dankserver
|
||||
metric_relabel_configs:
|
||||
- source_labels: [id]
|
||||
regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
|
||||
target_label: container
|
||||
replacement: '$1'
|
||||
- source_labels: [id]
|
||||
regex: '.*/\.service/.*'
|
||||
action: drop
|
||||
- source_labels: [container]
|
||||
regex: '.+'
|
||||
action: keep
|
||||
- source_labels: [cpu]
|
||||
regex: '^$|total'
|
||||
action: keep
|
||||
- regex: 'id|name|image|container_label.*|machine_id|boot_id'
|
||||
action: labeldrop
|
||||
Reference in new issue
Block a user