--- type: reference title: Fleet Audit and Posture description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work. tags: [audit, security, prometheus, caddy, monitoring, git] timestamp: 2026-10-05T00:00:00-07:00 --- # Fleet Audit and Posture ## Monitoring gap audit Use after any reboot, or whenever Prometheus targets look wrong. Two failure modes, only the first announces itself. **1. A service that should exist and doesn't is invisible.** A missing exporter shows up as nothing — no failing target if the job was configured for a host that no longer serves, no alert because the series is absent. So check **existence, not health**: ```bash podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones ls ~/.config/containers/systemd/*.container # what should be boot-persistent systemctl --user is-enabled .service # expect: generated ``` A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic case: rootless podman does not restart bare containers after the stop that shutdown performs, so a compose-only service dies at the first reboot and stays dead. If the compose file still names it, the repo's own rule is being violated — fix by writing the quadlet (mirror the same service on the other host), deleting the compose entry (one owner per container name), and making the deploy path reach it: planner service-set + path rule, dispatcher allowlist `:`, handler installing from `"$TARGET/..."` never a hardcoded peer's path. **2. Classify every down target — do not report a count.** `targets?state=active` gives health **and** last error; two down targets can mean opposite things: | Symptom | Meaning | |---|---| | `connection refused` | service not listening → real outage (podman ps, logs, exit codes) | | `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine | | `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter | | target absent | §1 | ```bash curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c " import json,sys for t in json.load(sys.stdin)['data']['activeTargets']: if t['health'] != 'up': print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])" curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http:///metrics # hand-probe before believing the scraper ``` "7 of 11 up" without the four reasons is how a real outage hid behind three config defects on this fleet — the crash-looping GPU service was one of four failures reported as a number. **3. Exposure posture while you are in there.** An exporter bound to the wildcard publishes an unauthenticated API on every interface. Expect exactly two listeners per host-exporter — loopback for host-local probes, the host's own LAN address for the Prometheus container: `ss -ltn | grep ` must show two explicit addresses, never `0.0.0.0`. **4. Leave decisions that are not yours.** Fix mechanical drift (missing unit, wrong path, wildcard bind). Do NOT disable a probe to green a dashboard, or "fix" a crash-looping service whose posture depends on owner intent — capture the evidence (restart count, exit code, last error line) and hand it over intact. ## vhost proxy health probes After deploying or auditing a Caddy/Anubis front door, verify **from a client**, not from the backend's localhost — that is where transport mismatches, auth walls, and dead upstreams show up. ``` curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:///ping curl -s -o /dev/null -m 6 -w "%{http_code}\n" https:/// ``` A status code is a diagnosis, not a failure: - `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping` bypasses auth by design) — probe `/` to decide. - `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404 there = healthy). - `502` = Caddy could not reach/negotiate with the upstream: dead backend **or transport mismatch**. Probe the backend port directly from the proxy's network (`ssh curl http://:/…`). Direct 200 + vhost 502 ⇒ config bug, not an outage. - Login page to anonymous (grep body for `password`) = auth ON; instant redirect to `/transferlist/`-style routes = auth OFF = open door. **Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it → 502 per request while the same port answers 200 on plain HTTP/1.1. Fix = drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain HTTP off-loopback and needs a TLS transport. Match the transport to what the backend speaks. **Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits **1 when zero mismatches** — inside an `&&` chain it silently kills everything after it while still printing `0`. Wrap: `(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`. ## Security posture audit (behavioral) Answers "what can the internet touch, and is everything else actually gated?" — **all findings must be behavioral probes**, not config reads. **1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or redirect-only. Public DNS: `dig +short +time=3 +tries=1 .franksdank.cloud @1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by DNS absence. **2. Guard chain (expect the challenge, not the app).** Per guarded vhost: `curl -sk https:///` and grep the body for `anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200 with NO markers means the app is served directly — the guard is bypassed. **3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` → 401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/` → 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/` → 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin `/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr `/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent `/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to `/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/` → 200 (flow HTML is React; grep "password" finds nothing — the 200 on the flow URL is the proof). **4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is unreliable; scan from gluetun's ProtonVPN egress: ```bash ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \ 8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \ 5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \ >/dev/null 2>&1 && echo OPEN: \$p; done"' ``` Confirm the egress is truly external first (`wget -qO- https://api.ipify.org` → ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports (443, 32400) are false negatives** — port open, protocol mismatch; re-probe with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it errors instantly and the `&&` chain silently skips the echo (a sub-second wall time on a "timeout" scan means the flag failed, not that ports closed). **5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live (loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds: `sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no auth (OpenAI API, no token). Stale-record check: current WAN IP = `ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress); compare against every public A record — a record at a different Comcast IP is stale. **Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`, `search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all verified) + `auth` (IdP, by design); everything else LAN-only. Open findings: LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP); `plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin 192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener (dead config, no exposure). ## Stash supersession audit before drop Use when a repo (especially a live-deployment checkout where the working tree IS the deployment) has an old stash whose content may have been superseded by later commits, and dropping blindly risks losing unique facts. 1. **Inventory**: `git stash show --stat stash@{0}`. 2. **Probe for unique content per file**: `git show "stash@{0}:"` and diff against HEAD; count lines present only in the stash version. A file with zero unique-vs-HEAD lines is fully superseded — safe to drop. 3. **Classify each unique-content hit**: *stale fact* (superseded by a later refactor, e.g. old model naming) → drop is correct; *live durable fact* (an operational invariant never re-recorded) → extract it into live docs/memory or flag to the user before dropping. 4. **Archive before dropping** — zero loss: tag the stash commit, then drop: ```bash git tag archive/stash---wip "stash@{0}" git stash drop "stash@{0}" ``` The tag pins the full stash object including its index parent; recoverable via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- ` for the untracked-files third parent. Stashes are invisible in normal history and expire from reflogs (~90 days); tagging converts "drop" into a reversible archive at zero workflow cost. ## Whole-fleet audit (git/test/live/docs cross-check) Deep-audit the homelab fleet — git/test state, live service health, and docs-vs-live cross-check. Run from the dev checkout (`/home/blank/Projects/serverAppConfig`). **1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT trust the display order of parallel SSH outputs: they can render crossed and you will attribute host A's output to host B (2026-09-21: a false host↔IP "conflict" dissolved by a single self-contained per-host probe). Put `hostname` + a distinctive marker + the LLM identity in the SAME command: ``` for ip in 192.168.0.143 192.168.0.235; do ssh -o BatchMode=yes dank@$ip \ 'echo "host=$(hostname)"; \ echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \ echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo' done ``` Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw = AI stack + `frank-dragoon` :7770 + `frank-zealot` :7771 (RTX 3090 + Strix Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-archon` :7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so it is unaffected by IP drift. **2. LLM endpoints — health + identity.** `curl /health` per endpoint, then `/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is the pass condition. **3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE fleet. **A single-test flake is normal**, not a regression: the plex crawl intermittently lands on `app.plex.tv`, the grafana SSO test occasionally times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just that spec (`npx playwright test --project=standard specs/-.spec.ts`) to classify before alarming. A `deploy.sh` e2e gate can fail on a flake while the change itself is fine — verify the actual change live, then re-run the suite. **4. Repo state.** `git status --short --branch; git log --oneline -20; git stash list; git worktree list` (prune stale worktrees), and `du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a **shared checkout with parallel instances**: uncommitted peer WIP and stash-list churn are expected. Never commit or drop a peer's files. To deploy with a dirty tree: `git stash push -u -m "..." -- `, deploy, `git stash pop`. `deploy.sh` refuses a dirty tree. **5. Prometheus — expected-down targets (not outages).** Query `/api/v1?query=up` from inside the container; the structurally red-by- construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape, `gitea` exporter off) are documented in [[Observability]] — anything else down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is a *host bind*, not a published port, so the bridge-networked Prometheus container cannot reach it at the host's LAN IP — netavark DNATs published ports only. Scrape same-host services via `host.containers.internal:`; verify the fix from inside the container before editing: `podman exec prometheus wget -qO- http://host.containers.internal:/metrics`. **6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔ stack mapping against step 1. The LLM endpoint table and `models.yml` are probed live and stay current; the prose host labels are the part that goes stale after an IP move. Report discrepancies; do not "fix" prose from a single crossed probe. ## Bulk-redact an identity from a docs + config tree Removing a retired host / machine / project name from a repo that holds both documentation and live config. Evidence-first, surgical, reversible. **1. Inventory before touching anything.** Count hits **per file** (the count decides whether this is a 5-minute job or a user-input decision), excluding logs/`*.db`/runtime state, and capture `git status --short` first — you must know which files carry in-flight peer WIP so you never stage them. **2. Classify every hit:** | Bucket | Test | Action | |---|---|---| | **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") | | **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it | | **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field | Load-bearing examples that look like prose but are not: `install.sh` doing `PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest *filename* is the key); a `--host ` case list; a Wi-Fi SSID. Hardware model strings, driver versions, PCI addresses are subject matter, not identity. State the boundary in the commit message. **3. Subdirectories: distill, don't rename or keep.** For a per-host doc tree (`docs/system//`), ask before acting — rename/trash/distill are genuinely different. When distilling: read every page, sort into still-true-and-machine-independent (fold into a general page) vs specific-to-the-retired-box (let go), then `mv` the directory to `.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by filename anywhere in the tree, so deleting a page can break a link in a file you never grepped — run the link checker. **4. Execute with count assertions, never blind sed.** Per-file mapping of `(path, old_exact_string, new_string, expected_count)`; abort the entry if the count differs; exact strings, not regex. **Do not hand this to a subagent** — observed: dropped list markers so bullets stopped rendering, a numbered list with two `3.` items, "corrected" paths to nonexistent directories, a deleted sentence where a reword was required. If a subagent already did it: `git checkout --` its touched paths (excluding the WIP files from step 1) and redo it. **5. Audit:** `grep -rniE '|' ` (expect nothing), then `git diff -U0 -- | grep -E '^\+[^+]'` and **read every added line** — the diff is the step that catches damage; the grep only proves a string is gone. Then the repo's own linter. **6. Commit scoped** — only your own paths, never `git add -A`. If the checkout IS the live deployment, committing on the current branch is usually correct; a branch switch can swap live config out from under a running process. Push/PR only when asked. **7. Report the residue honestly** — what still carries the name and why: excluded scopes, load-bearing config, git history, and the cost of rewriting history (force-push + re-clones). "Purged from the working tree" is a true claim; "purged" alone is not. ## Related - [[Observability]] — the monitored stack - [[Stale Signal Audit]] — the down-target decision table - [[Observability Provisioning]] — adding scrape targets and rules - [[Homepage Status Probes]] — the status page that renders probe states - [[Deploy State Triage]] — the deploy plumbing the audits exercise