serverAppConfigs/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md

---
type: reference
title: Fleet Audit and Posture
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
tags: [audit, security, prometheus, caddy, monitoring, git]
timestamp: 2026-10-05T00:00:00-07:00
---

# Fleet Audit and Posture

## Monitoring gap audit

Use after any reboot, or whenever Prometheus targets look wrong. Two failure
modes, only the first announces itself.

**1. A service that should exist and doesn't is invisible.** A missing
exporter shows up as nothing — no failing target if the job was configured
for a host that no longer serves, no alert because the series is absent. So
check **existence, not health**:

```bash
podman ps -a --format '{{.Names}} {{.Status}}'   # includes dead ones
ls ~/.config/containers/systemd/*.container      # what should be boot-persistent
systemctl --user is-enabled <name>.service       # expect: generated
```

A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
case: rootless podman does not restart bare containers after the stop that
shutdown performs, so a compose-only service dies at the first reboot and
stays dead. If the compose file still names it, the repo's own rule is being
violated — fix by writing the quadlet (mirror the same service on the other
host), deleting the compose entry (one owner per container name), and making
the deploy path reach it: planner service-set + path rule, dispatcher
allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
a hardcoded peer's path.

**2. Classify every down target — do not report a count.**
`targets?state=active` gives health **and** last error; two down targets can
mean opposite things:

| Symptom | Meaning |
|---|---|
| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
| target absent | §1 |

```bash
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)['data']['activeTargets']:
    if t['health'] != 'up':
        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics   # hand-probe before believing the scraper
```

"7 of 11 up" without the four reasons is how a real outage hid behind three
config defects on this fleet — the crash-looping GPU service was one of four
failures reported as a number.

**3. Exposure posture while you are in there.** An exporter bound to the
wildcard publishes an unauthenticated API on every interface. Expect exactly
two listeners per host-exporter — loopback for host-local probes, the host's
own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
show two explicit addresses, never `0.0.0.0`.

**4. Leave decisions that are not yours.** Fix mechanical drift (missing
unit, wrong path, wildcard bind). Do NOT disable a probe to green a
dashboard, or "fix" a crash-looping service whose posture depends on owner
intent — capture the evidence (restart count, exit code, last error line) and
hand it over intact.

## vhost proxy health probes

After deploying or auditing a Caddy/Anubis front door, verify **from a
client**, not from the backend's localhost — that is where transport
mismatches, auth walls, and dead upstreams show up.

```
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
```

A status code is a diagnosis, not a failure:

- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
  bypasses auth by design) — probe `/` to decide.
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
  there = healthy).
- `502` = Caddy could not reach/negotiate with the upstream: dead backend
  **or transport mismatch**. Probe the backend port directly from the proxy's
  network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
  vhost 502 ⇒ config bug, not an outage.
- Login page to anonymous (grep body for `password`) = auth ON; instant
  redirect to `/transferlist/`-style routes = auth OFF = open door.

**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
HTTP off-loopback and needs a TLS transport. Match the transport to what the
backend speaks.

**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
**1 when zero mismatches** — inside an `&&` chain it silently kills
everything after it while still printing `0`. Wrap:
`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.

## Security posture audit (behavioral)

Answers "what can the internet touch, and is everything else actually
gated?" — **all findings must be behavioral probes**, not config reads.

**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
DNS absence.

**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
`curl -sk https://<vhost>/` and grep the body for
`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
with NO markers means the app is served directly — the guard is bypassed.

**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
flow URL is the proof).

**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
unreliable; scan from gluetun's ProtonVPN egress:

```bash
ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
  8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
  5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
  >/dev/null 2>&1 && echo OPEN: \$p; done"'
```

Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
errors instantly and the `&&` chain silently skips the echo (a sub-second
wall time on a "timeout" scan means the flag failed, not that ports closed).

**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
auth (OpenAI API, no token). Stale-record check: current WAN IP =
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
compare against every public A record — a record at a different Comcast IP
is stale.

**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
(dead config, no exposure).

## Stash supersession audit before drop

Use when a repo (especially a live-deployment checkout where the working tree
IS the deployment) has an old stash whose content may have been superseded by
later commits, and dropping blindly risks losing unique facts.

1. **Inventory**: `git stash show --stat stash@{0}`.
2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
   diff against HEAD; count lines present only in the stash version. A file
   with zero unique-vs-HEAD lines is fully superseded — safe to drop.
3. **Classify each unique-content hit**: *stale fact* (superseded by a later
   refactor, e.g. old model naming) → drop is correct; *live durable fact*
   (an operational invariant never re-recorded) → extract it into live
   docs/memory or flag to the user before dropping.
4. **Archive before dropping** — zero loss: tag the stash commit, then drop:

```bash
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
git stash drop "stash@{0}"
```

The tag pins the full stash object including its index parent; recoverable
via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
for the untracked-files third parent. Stashes are invisible in normal history
and expire from reflogs (~90 days); tagging converts "drop" into a
reversible archive at zero workflow cost.

## Whole-fleet audit (git/test/live/docs cross-check)

Deep-audit the homelab fleet — git/test state, live service health, and
docs-vs-live cross-check. Run from the dev checkout
(`/home/blank/Projects/serverAppConfig`).

**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
trust the display order of parallel SSH outputs: they can render crossed and
you will attribute host A's output to host B (2026-09-21: a false host↔IP
"conflict" dissolved by a single self-contained per-host probe). Put
`hostname` + a distinctive marker + the LLM identity in the SAME command:

```
for ip in 192.168.0.143 192.168.0.235; do
  ssh -o BatchMode=yes dank@$ip \
    'echo "host=$(hostname)"; \
     echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
     echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
done
```

Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
= AI stack + `frank-dragoon` :7770 + `frank-zealot` :7771 (RTX 3090 + Strix
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-archon`
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
it is unaffected by IP drift.

**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
the pass condition.

**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
fleet. **A single-test flake is normal**, not a regression: the plex crawl
intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
while the change itself is fine — verify the actual change live, then re-run
the suite.

**4. Repo state.** `git status --short --branch; git log --oneline -20;
git stash list; git worktree list` (prune stale worktrees), and
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
**shared checkout with parallel instances**: uncommitted peer WIP and
stash-list churn are expected. Never commit or drop a peer's files. To
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.

**5. Prometheus — expected-down targets (not outages).** Query
`/api/v1?query=up` from inside the container; the structurally red-by-
construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
`gitea` exporter off) are documented in [[Observability]] — anything else
down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
a *host bind*, not a published port, so the bridge-networked Prometheus
container cannot reach it at the host's LAN IP — netavark DNATs published
ports only. Scrape same-host services via
`host.containers.internal:<port>`; verify the fix from inside the container
before editing: `podman exec prometheus wget -qO-
http://host.containers.internal:<port>/metrics`.

**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
stack mapping against step 1. The LLM endpoint table and `models.yml` are
probed live and stay current; the prose host labels are the part that goes
stale after an IP move. Report discrepancies; do not "fix" prose from a
single crossed probe.

## Bulk-redact an identity from a docs + config tree

Removing a retired host / machine / project name from a repo that holds both
documentation and live config. Evidence-first, surgical, reversible.

**1. Inventory before touching anything.** Count hits **per file** (the
count decides whether this is a 5-minute job or a user-input decision),
excluding logs/`*.db`/runtime state, and capture `git status --short` first
— you must know which files carry in-flight peer WIP so you never stage
them.

**2. Classify every hit:**

| Bucket | Test | Action |
|---|---|---|
| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |

Load-bearing examples that look like prose but are not: `install.sh` doing
`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
model strings, driver versions, PCI addresses are subject matter, not
identity. State the boundary in the commit message.

**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
genuinely different. When distilling: read every page, sort into
still-true-and-machine-independent (fold into a general page) vs
specific-to-the-retired-box (let go), then `mv` the directory to
`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
filename anywhere in the tree, so deleting a page can break a link in a
file you never grepped — run the link checker.

**4. Execute with count assertions, never blind sed.** Per-file mapping of
`(path, old_exact_string, new_string, expected_count)`; abort the entry if
the count differs; exact strings, not regex. **Do not hand this to a
subagent** — observed: dropped list markers so bullets stopped rendering, a
numbered list with two `3.` items, "corrected" paths to nonexistent
directories, a deleted sentence where a reword was required. If a subagent
already did it: `git checkout --` its touched paths (excluding the WIP
files from step 1) and redo it.

**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
— the diff is the step that catches damage; the grep only proves a string
is gone. Then the repo's own linter.

**6. Commit scoped** — only your own paths, never `git add -A`. If the
checkout IS the live deployment, committing on the current branch is usually
correct; a branch switch can swap live config out from under a running
process. Push/PR only when asked.

**7. Report the residue honestly** — what still carries the name and why:
excluded scopes, load-bearing config, git history, and the cost of rewriting
history (force-push + re-clones). "Purged from the working tree" is a true
claim; "purged" alone is not.

## Related

- [[Observability]] — the monitored stack
- [[Stale Signal Audit]] — the down-target decision table
- [[Observability Provisioning]] — adding scrape targets and rules
- [[Homepage Status Probes]] — the status page that renders probe states
- [[Deploy State Triage]] — the deploy plumbing the audits exercise