agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: fleet-monitoring-gap-audit
|
||||
description: "Audit a git-declared homelab fleet for monitoring gaps: find services that should exist but silently don't (compose-only drift), classify each down target as outage vs probe defect, and check exporter bind addresses."
|
||||
---
|
||||
|
||||
# Auditing a homelab fleet's monitoring for what *should* be there
|
||||
|
||||
Use after any reboot, or whenever Prometheus/Targets looks wrong on a fleet whose
|
||||
services are declared in a git repo. The two failure modes this catches are
|
||||
different, and only the first announces itself.
|
||||
|
||||
## 1. A service that should exist and doesn't is invisible
|
||||
|
||||
A missing exporter shows up as **nothing**: no failing target if the job was
|
||||
configured for a host that no longer serves, no alert because the series is
|
||||
simply absent. So check existence, not health:
|
||||
|
||||
```bash
|
||||
# Every container the host actually runs, including dead ones
|
||||
podman ps -a --format '{{.Names}} {{.Status}}'
|
||||
# Every container that is supposed to be boot-persistent
|
||||
ls ~/.config/containers/systemd/*.container
|
||||
systemctl --user is-enabled <name>.service # expect: generated
|
||||
```
|
||||
|
||||
A container in `ps -a` that is `Exited (…)` **and has no unit** is the classic
|
||||
case: rootless Podman does not restart bare containers after the stop that
|
||||
shutdown performs, so a compose-only service dies at the first reboot and stays
|
||||
dead. Check the compose file for the name — if it is there and has no unit, the
|
||||
repo's own rule is being violated. Fix by writing the unit (mirror the same
|
||||
service on the other host), deleting the compose entry (one owner per container
|
||||
name), and making sure the deploy path can reach it:
|
||||
|
||||
- planner: service in the target's service set + a path rule for its directory
|
||||
- dispatcher allowlist: `<target>:<service>`
|
||||
- handler: install from `"$TARGET/..."`, never a hardcoded peer's path
|
||||
|
||||
## 2. Classify every down target — do not report a count
|
||||
|
||||
`targets?state=active` gives health **and** the last error. Two down targets can
|
||||
mean opposite things:
|
||||
|
||||
| Symptom | Meaning |
|
||||
|---|---|
|
||||
| `connection refused` | the service is not listening → real outage (check `podman ps`, logs, exit codes) |
|
||||
| `404` on `/metrics` | the exporter path is wrong or metrics are disabled → config defect, service fine |
|
||||
| `unsupported Content-Type …` | you are scraping an HTML *page* as if it were metrics → the probe is inert; Prometheus 3 refuses HTML. A "blackbox-style probe" needs a blackbox exporter |
|
||||
| target absent entirely | see §1 |
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
|
||||
import json,sys
|
||||
for t in json.load(sys.stdin)['data']['activeTargets']:
|
||||
if t['health'] != 'up':
|
||||
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
|
||||
# and probe each by hand before believing the scraper
|
||||
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics
|
||||
```
|
||||
|
||||
Say which are down and why. "7 of 11 up" without the four reasons is how a real
|
||||
outage hides behind three config defects — that is exactly what one fleet did:
|
||||
its crash-looping GPU service was one of four failures reported as a number.
|
||||
|
||||
## 3. Check the exposure posture while you are in there
|
||||
|
||||
An exporter that binds the wildcard publishes an unauthenticated API on every
|
||||
interface. Expect exactly two listeners per host-exporter — loopback for
|
||||
host-local probes, the host's own LAN address for the Prometheus container:
|
||||
|
||||
```bash
|
||||
ss -ltn | grep <port> # must show the two explicit addresses, never 0.0.0.0
|
||||
```
|
||||
|
||||
## 4. Leave the decisions that are not yours
|
||||
|
||||
Fix mechanical drift (missing unit, wrong path, wildcard bind). Do **not**:
|
||||
disable a probe to make a dashboard green, or "fix" a crash-looping service whose
|
||||
correct posture depends on what the owner wants — capture the evidence (restart
|
||||
count, exit code, last error line) and hand it over intact.
|
||||
Reference in new issue
Block a user