command-center/agent/managed-skills/fleet-monitoring-gap-audit/SKILL.md

name, description

name	description
fleet-monitoring-gap-audit	Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses.

Auditing a homelab fleet's monitoring for what should be there

Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose services are declared in a git repo. The two failure modes this catches are different, and only the first announces itself.

1. A service that should exist and doesn't is invisible

A missing exporter shows up as nothing: no failing target if the job was configured for a host that no longer serves, no alert because the series is simply absent. So check existence, not health:

# Every container the host actually runs, including dead ones
podman ps -a --format '{{.Names}} {{.Status}}'
# Every container that is supposed to be boot-persistent
ls ~/.config/containers/systemd/*.container
systemctl --user is-enabled <name>.service   # expect: generated

A container in ps -a that is Exited (…) and has no unit is the classic case: rootless Podman does not restart bare containers after the stop that shutdown performs, so a compose-only service dies at the first reboot and stays dead. Check the compose file for the name — if it is there and has no unit, the repo's own rule is being violated. Fix by writing the unit (mirror the same service on the other host), deleting the compose entry (one owner per container name), and making sure the deploy path can reach it:

• planner: service in the target's service set + a path rule for its directory
• dispatcher allowlist: <target>:<service>
• handler: install from "$TARGET/...", never a hardcoded peer's path

2. Classify every down target — do not report a count

targets?state=active gives health and the last error. Two down targets can mean opposite things:

Symptom	Meaning
connection refused	the service is not listening → real outage (check podman ps, logs, exit codes)
404 on /metrics	the exporter path is wrong or metrics are disabled → config defect, service fine
unsupported Content-Type …	you are scraping an HTML page as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter
target absent entirely	see §1

curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)['data']['activeTargets']:
    if t['health'] != 'up':
        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
# and probe each by hand before believing the scraper
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics

Say which are down and why. "7 of 11 up" without the four reasons is how a real outage hides behind three config defects — that is exactly what one fleet did: its crash-looping GPU service was one of four failures reported as a number.

3. Check the exposure posture while you are in there

An exporter that binds the wildcard publishes an unauthenticated API on every interface. Expect exactly two listeners per host-exporter — loopback for host-local probes, the host's own LAN address for the Prometheus container:

ss -ltn | grep <port>    # must show the two explicit addresses, never 0.0.0.0

4. Leave the decisions that are not yours

Fix mechanical drift (missing unit, wrong path, wildcard bind). Do not: disable a probe to make a dashboard green, or "fix" a crash-looping service whose correct posture depends on what the owner wants — capture the evidence (restart count, exit code, last error line) and hand it over intact.