Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
80 lines
3.7 KiB
Markdown
80 lines
3.7 KiB
Markdown
---
|
|
name: fleet-monitoring-gap-audit
|
|
description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
|
|
---
|
|
|
|
# Auditing a homelab fleet's monitoring for what *should* be there
|
|
|
|
Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
|
|
services are declared in a git repo. The two failure modes this catches are
|
|
different, and only the first announces itself.
|
|
|
|
## 1. A service that should exist and doesn't is invisible
|
|
|
|
A missing exporter shows up as **nothing**: no failing target if the job was
|
|
configured for a host that no longer serves, no alert because the series is
|
|
simply absent. So check existence, not health:
|
|
|
|
```bash
|
|
# Every container the host actually runs, including dead ones
|
|
podman ps -a --format '{{.Names}} {{.Status}}'
|
|
# Every container that is supposed to be boot-persistent
|
|
ls ~/.config/containers/systemd/*.container
|
|
systemctl --user is-enabled <name>.service # expect: generated
|
|
```
|
|
|
|
A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
|
|
case: rootless Podman does not restart bare containers after the stop that
|
|
shutdown performs, so a compose-only service dies at the first reboot and stays
|
|
dead. Check the compose file for the name — if it is there and has no unit, the
|
|
repo's own rule is being violated. Fix by writing the unit (mirror the same
|
|
service on the other host), deleting the compose entry (one owner per container
|
|
name), and making sure the deploy path can reach it:
|
|
|
|
- planner: service in the target's service set + a path rule for its directory
|
|
- dispatcher allowlist: `<target>:<service>`
|
|
- handler: install from `"$TARGET/..."`, never a hardcoded peer's path
|
|
|
|
## 2. Classify every down target — do not report a count
|
|
|
|
`targets?state=active` gives health **and** the last error. Two down targets can
|
|
mean opposite things:
|
|
|
|
| Symptom | Meaning |
|
|
|---|---|
|
|
| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
|
|
| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
|
|
| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
|
|
| target absent entirely | see §1 |
|
|
|
|
```bash
|
|
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
|
import json,sys
|
|
for t in json.load(sys.stdin)['data']['activeTargets']:
|
|
if t['health'] != 'up':
|
|
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
|
# and probe each by hand before believing the scraper
|
|
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
|
|
```
|
|
|
|
Say which are down and why. "7 of 11 up" without the four reasons is how a real
|
|
outage hides behind three config defects — that is exactly what one fleet did:
|
|
its crash-looping GPU service was one of four failures reported as a number.
|
|
|
|
## 3. Check the exposure posture while you are in there
|
|
|
|
An exporter that binds the wildcard publishes an unauthenticated API on every
|
|
interface. Expect exactly two listeners per host-exporter — loopback for
|
|
host-local probes, the host's own LAN address for the Prometheus container:
|
|
|
|
```bash
|
|
ss -ltn | grep <port> # must show the two explicit addresses, never 0.0.0.0
|
|
```
|
|
|
|
## 4. Leave the decisions that are not yours
|
|
|
|
Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
|
|
disable a probe to make a dashboard green, or "fix" a crash-looping service whose
|
|
correct posture depends on what the owner wants — capture the evidence (restart
|
|
count, exit code, last error line) and hand it over intact.
|