---
name: fleet-monitoring-gap-audit
description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
---

# Auditing a homelab fleet's monitoring for what *should* be there

Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
services are declared in a git repo. The two failure modes this catches are
different, and only the first announces itself.

## 1. A service that should exist and doesn't is invisible

A missing exporter shows up as **nothing**: no failing target if the job was
configured for a host that no longer serves, no alert because the series is
simply absent. So check existence, not health:

```bash
# Every container the host actually runs, including dead ones
podman ps -a --format '{{.Names}} {{.Status}}'
# Every container that is supposed to be boot-persistent
ls ~/.config/containers/systemd/*.container
systemctl --user is-enabled <name>.service   # expect: generated
```

A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
case: rootless Podman does not restart bare containers after the stop that
shutdown performs, so a compose-only service dies at the first reboot and stays
dead. Check the compose file for the name — if it is there and has no unit, the
repo's own rule is being violated. Fix by writing the unit (mirror the same
service on the other host), deleting the compose entry (one owner per container
name), and making sure the deploy path can reach it:

- planner: service in the target's service set + a path rule for its directory
- dispatcher allowlist: `<target>:<service>`
- handler: install from `"$TARGET/..."`, never a hardcoded peer's path

## 2. Classify every down target — do not report a count

`targets?state=active` gives health **and** the last error. Two down targets can
mean opposite things:

| Symptom | Meaning |
|---|---|
| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
| target absent entirely | see §1 |

```bash
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)['data']['activeTargets']:
    if t['health'] != 'up':
        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
# and probe each by hand before believing the scraper
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
```

Say which are down and why. "7 of 11 up" without the four reasons is how a real
outage hides behind three config defects — that is exactly what one fleet did:
its crash-looping GPU service was one of four failures reported as a number.

## 3. Check the exposure posture while you are in there

An exporter that binds the wildcard publishes an unauthenticated API on every
interface. Expect exactly two listeners per host-exporter — loopback for
host-local probes, the host's own LAN address for the Prometheus container:

```bash
ss -ltn | grep <port>    # must show the two explicit addresses, never 0.0.0.0
```

## 4. Leave the decisions that are not yours

Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
disable a probe to make a dashboard green, or "fix" a crash-looping service whose
correct posture depends on what the owner wants — capture the evidence (restart
count, exit code, last error line) and hand it over intact.