serverAppConfigs/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Fleet Audit and Posture	Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture.	audit security prometheus caddy monitoring git	2026-10-05T00:00:00-07:00

Fleet Audit and Posture

Monitoring gap audit

Use after any reboot, or whenever Prometheus targets look wrong. Two failure modes, only the first announces itself.

1. A service that should exist and doesn't is invisible. A missing exporter shows up as nothing — no failing target if the job was configured for a host that no longer serves, no alert because the series is absent. So check existence, not health:

podman ps -a --format '{{.Names}} {{.Status}}'   # includes dead ones
ls ~/.config/containers/systemd/*.container      # what should be boot-persistent
systemctl --user is-enabled <name>.service       # expect: generated

A container in ps -a that is Exited(…) and has no unit is the classic case: rootless podman does not restart bare containers after the stop that shutdown performs, so a compose-only service dies at the first reboot and stays dead. If the compose file still names it, the repo's own rule is being violated — fix by writing the quadlet (mirror the same service on the other host), deleting the compose entry (one owner per container name), and making the deploy path reach it: planner service-set + path rule, dispatcher allowlist <target>:<service>, handler installing from "$TARGET/..." never a hardcoded peer's path.

2. Classify every down target — do not report a count. targets?state=active gives health and last error; two down targets can mean opposite things:

Symptom	Meaning
connection refused	service not listening → real outage (podman ps, logs, exit codes)
404 on /metrics	exporter path wrong or metrics disabled → config defect, service fine
unsupported Content-Type …	scraping an HTML page as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter
target absent	§1

curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)['data']['activeTargets']:
    if t['health'] != 'up':
        print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics   # hand-probe before believing the scraper

"7 of 11 up" without the four reasons is how a real outage hid behind three config defects on this fleet — the crash-looping GPU service was one of four failures reported as a number.

3. Exposure posture while you are in there. An exporter bound to the wildcard publishes an unauthenticated API on every interface. Expect exactly two listeners per host-exporter — loopback for host-local probes, the host's own LAN address for the Prometheus container: ss -ltn | grep <port> must show two explicit addresses, never 0.0.0.0.

4. Leave decisions that are not yours. Fix mechanical drift (missing unit, wrong path, wildcard bind). Do NOT disable a probe to green a dashboard, or "fix" a crash-looping service whose posture depends on owner intent — capture the evidence (restart count, exit code, last error line) and hand it over intact.

vhost proxy health probes

After deploying or auditing a Caddy/Anubis front door, verify from a client, not from the backend's localhost — that is where transport mismatches, auth walls, and dead upstreams show up.

curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/

A status code is a diagnosis, not a failure:

• 200 on /ping from *arrs = alive; auth state unknown (/ping bypasses auth by design) — probe / to decide.
• 404 = backend alive, route missing (qBittorrent has no /ping; 404 there = healthy).
• 502 = Caddy could not reach/negotiate with the upstream: dead backend or transport mismatch. Probe the backend port directly from the proxy's network (ssh <proxy-host> curl http://<backend>:<port>/…). Direct 200 + vhost 502 ⇒ config bug, not an outage.
• Login page to anonymous (grep body for password) = auth ON; instant redirect to /transferlist/-style routes = auth OFF = open door.

Transport-mismatch mechanism: transport http {versions h2c 1.1} makes Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it → 502 per request while the same port answers 200 on plain HTTP/1.1. Fix = drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain HTTP off-loopback and needs a TLS transport. Match the transport to what the backend speaks.

Shell trap when gating commits on a test suite: grep -c "^not ok" exits 1 when zero mismatches — inside an && chain it silently kills everything after it while still printing 0. Wrap: (./tests/run.sh | grep -cE '^not ok' || true) && git commit ….

Security posture audit (behavioral)

Answers "what can the internet touch, and is everything else actually gated?" — all findings must be behavioral probes, not config reads.

1. Inventory (source of truth). dankclaw/Caddyfile: classify every vhost — guarded (reverse_proxy anubis:8923), exempt (direct upstream), or redirect-only. Public DNS: dig +short +time=3 +tries=1 <name>.franksdank.cloud @1.1.1.1 per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by DNS absence.

2. Guard chain (expect the challenge, not the app). Per guarded vhost: curl -sk https://<vhost>/ and grep the body for anubis|challenge|proof-of-work (~8 markers on a live challenge page). A 200 with NO markers means the app is served directly — the guard is bypassed.

3. Exempt vhosts — per-app gating probes. OpenWebUI /api/v1/users/me → 401. Gitea /api/v1/user → 401; repo visibility /api/v1/repos/dank/<repo> → 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex /web/ → 302 (login is JS-rendered). Calibre-Web / → 302 to /login. Jellyfin /Users → 401. Prowlarr/Sonarr/Radarr /api/v3/system/status; Lidarr/Readarr /api/v1/system/status → 401 = gated, 200 = fully open. qBittorrent /api/v2/torrents/info → 403 without SID. Hermes/hermesadmin / → 302 to /login. IdP (auth.): / → 302 → /if/flow/default-authentication-flow/ → 200 (flow HTML is React; grep "password" finds nothing — the 200 on the flow URL is the proof).

4. WAN port scan — from an EXTERNAL vantage. LAN→WAN hairpin is unreliable; scan from gluetun's ProtonVPN egress:

ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
  8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
  5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
  >/dev/null 2>&1 && echo OPEN: \$p; done"'

Confirm the egress is truly external first (wget -qO- https://api.ipify.org → ProtonVPN IP, not Comcast). Trap: plain-HTTP probes against TLS ports (443, 32400) are false negatives — port open, protocol mismatch; re-probe with wget -qO- https://IP[:port]/. busybox wget has no --secure flag; it errors instantly and the && chain silently skips the echo (a sub-second wall time on a "timeout" scan means the flag failed, not that ports closed).

5. Residuals. Grafana basic-auth residual: curl -u admin:admin http://127.0.0.1:3002/api/org → 200 means admin:admin still live (loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds: sudo ss -tln | grep -E ':(7770|7771)' — 0.0.0.0 = LAN-reachable with no auth (OpenAI API, no token). Stale-record check: current WAN IP = ssh frankbook-pro 'curl -s https://api.ipify.org' (residential egress) compare against every public A record — a record at a different Comcast IP is stale.

Baseline (2026-09-21): WAN: only 80 + 443 open. Public names: www search (guarded) + ai, gitea, plex, books, jelly (own auth, all verified) + auth (IdP, by design); everything else LAN-only. Open findings: LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana admin:admin loopback residual; plex.franksdank.casa stale (73.59.75.210 = previous Comcast IP) plex.franksdank.cloud WAN-broken (Plex Web 302s to LAN origin 192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener (dead config, no exposure).

Stash supersession audit before drop

Use when a repo (especially a live-deployment checkout where the working tree IS the deployment) has an old stash whose content may have been superseded by later commits, and dropping blindly risks losing unique facts.

1. Inventory: git stash show --stat stash@{0}.
2. Probe for unique content per file: git show "stash@{0}:<path>" and diff against HEAD; count lines present only in the stash version. A file with zero unique-vs-HEAD lines is fully superseded — safe to drop.
3. Classify each unique-content hit: stale fact (superseded by a later refactor, e.g. old model naming) → drop is correct; live durable fact (an operational invariant never re-recorded) → extract it into live docs/memory or flag to the user before dropping.
4. Archive before dropping — zero loss: tag the stash commit, then drop:

git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
git stash drop "stash@{0}"

The tag pins the full stash object including its index parent; recoverable via git show archive/stash-… or git checkout archive/stash-…^3 -- <path> for the untracked-files third parent. Stashes are invisible in normal history and expire from reflogs (~90 days); tagging converts "drop" into a reversible archive at zero workflow cost.

Whole-fleet audit (git/test/live/docs cross-check)

Deep-audit the homelab fleet — git/test state, live service health, and docs-vs-live cross-check. Run from the dev checkout (/home/blank/Projects/serverAppConfig).

1. Probe the live fleet — self-identifying, ONE ssh per host. Do NOT trust the display order of parallel SSH outputs: they can render crossed and you will attribute host A's output to host B (2026-09-21: a false host↔IP "conflict" dissolved by a single self-contained per-host probe). Put hostname + a distinctive marker + the LLM identity in the SAME command:

for ip in 192.168.0.143 192.168.0.235; do
  ssh -o BatchMode=yes dank@$ip \
    'echo "host=$(hostname)"; \
     echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
     echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
done

Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw = AI stack + frank-dragoon :7770 + frank-zealot :7771 (RTX 3090 + Strix Halo iGPU); 192.168.0.235 = dankserver = media stack + frank-archon :7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is hostname-anchored (ssh dankclaw/ssh dankserver via ~/.ssh/config), so it is unaffected by IP drift.

2. LLM endpoints — health + identity. curl /health per endpoint, then /v1/models (first 120 chars) — a 200 with the right model id/name is the pass condition.

3. Tests — the deploy completeness gate. ./tests/run.sh (bats) must end ALL TESTS PASSED; tests/e2e/run.sh runs Playwright against the LIVE fleet. A single-test flake is normal, not a regression: the plex crawl intermittently lands on app.plex.tv, the grafana SSO test occasionally times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just that spec (npx playwright test --project=standard specs/<n>-<name>.spec.ts) to classify before alarming. A deploy.sh e2e gate can fail on a flake while the change itself is fine — verify the actual change live, then re-run the suite.

4. Repo state. git status --short --branch; git log --oneline -20; git stash list; git worktree list (prune stale worktrees), and du -sh .AGENTTRASH for staged deletions awaiting purge approval. This is a shared checkout with parallel instances: uncommitted peer WIP and stash-list churn are expected. Never commit or drop a peer's files. To deploy with a dirty tree: git stash push -u -m "..." -- <peer paths> deploy, git stash pop. deploy.sh refuses a dirty tree.

5. Prometheus — expected-down targets (not outages). Query /api/v1?query=up from inside the container; the structurally red-by- construction jobs (comfyui HTML scrape, services up/down-as-scrape, gitea exporter off) are documented in Observability — anything else down, especially a node-* or LLM job, is a real gap. The recurring trap: a same-host net=host bind (node-exporter :9100, llama-server :7770/:7771) is a host bind, not a published port, so the bridge-networked Prometheus container cannot reach it at the host's LAN IP — netavark DNATs published ports only. Scrape same-host services host.containers.internal:<port>; verify the fix from inside the container before editing: podman exec prometheus wget -qO- http://host.containers.internal:<port>/metrics.

6. Cross-check docs vs live. Confirm the AGENTS.md host↔IP↔hardware↔ stack mapping against step 1. The LLM endpoint table and models.yml are probed live and stay current; the prose host labels are the part that goes stale after an IP move. Report discrepancies; do not "fix" prose from a single crossed probe.

Bulk-redact an identity from a docs + config tree

Removing a retired host / machine / project name from a repo that holds both documentation and live config. Evidence-first, surgical, reversible.

1. Inventory before touching anything. Count hits per file (the count decides whether this is a 5-minute job or a user-input decision) excluding logs/*.db/runtime state, and capture git status --short first — you must know which files carry in-flight peer WIP so you never stage them.

2. Classify every hit:

Bucket	Test	Action
Narrative	prose describing the thing	Reword to the class ("the workstation", "a client box")
Load-bearing	the identifier IS a lookup key, hostname match, or config value	Keep. Prove it: read the code that consumes it
Evidence	quoted logs, transcripts, reports	Keep timestamps/lines; de-identify only the host field

Load-bearing examples that look like prose but are not: install.sh doing PKGS=("common.pkgs" "$HOST.pkgs") with HOST="$(hostname)" (the manifest filename is the key); a --host <name> case list; a Wi-Fi SSID. Hardware model strings, driver versions, PCI addresses are subject matter, not identity. State the boundary in the commit message.

3. Subdirectories: distill, don't rename or keep. For a per-host doc tree (docs/system/<host>/), ask before acting — rename/trash/distill are genuinely different. When distilling: read every page, sort into still-true-and-machine-independent (fold into a general page) vs specific-to-the-retired-box (let go), then mv the directory to .AGENTTRASH/YYYYMMDD-topic/ and repoint every index. Wikilinks resolve by filename anywhere in the tree, so deleting a page can break a link in a file you never grepped — run the link checker.

4. Execute with count assertions, never blind sed. Per-file mapping of (path, old_exact_string, new_string, expected_count); abort the entry if the count differs; exact strings, not regex. Do not hand this to a subagent — observed: dropped list markers so bullets stopped rendering, a numbered list with two 3. items, "corrected" paths to nonexistent directories, a deleted sentence where a reword was required. If a subagent already did it: git checkout -- its touched paths (excluding the WIP files from step 1) and redo it.

5. Audit: grep -rniE '<name>|<its-ip>' <scope> (expect nothing), then git diff -U0 -- <scope> | grep -E '^\+[^+]' and read every added line — the diff is the step that catches damage; the grep only proves a string is gone. Then the repo's own linter.

6. Commit scoped — only your own paths, never git add -A. If the checkout IS the live deployment, committing on the current branch is usually correct; a branch switch can swap live config out from under a running process. Push/PR only when asked.

7. Report the residue honestly — what still carries the name and why: excluded scopes, load-bearing config, git history, and the cost of rewriting history (force-push + re-clones). "Purged from the working tree" is a true claim; "purged" alone is not.

Related

• Observability — the monitored stack
• Stale Signal Audit — the down-target decision table
• Observability Provisioning — adding scrape targets and rules
• Homepage Status Probes — the status page that renders probe states
• Deploy State Triage — the deploy plumbing the audits exercise