dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: distill the managed skill library into the vault (waves 1-7)

Browse Source 
All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank committed 2026-10-05 21:19:15 -07:00
1 parent 8618ec6dbe
commit 33eae71d33
43 files changed
+5436 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Operations/Fleet Audit and Posture.md
+331
View File
@@ -0,0 +1,331 @@
---
type: reference
title: Fleet Audit and Posture
description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work.
tags: [audit, security, prometheus, caddy, monitoring, git]
timestamp: 2026-10-05T00:00:00-07:00
---
# Fleet Audit and Posture
## Monitoring gap audit
Use after any reboot, or whenever Prometheus targets look wrong. Two failure
modes, only the first announces itself.
**1. A service that should exist and doesn't is invisible.** A missing
exporter shows up as nothing — no failing target if the job was configured
for a host that no longer serves, no alert because the series is absent. So
check **existence, not health**:
```bash
podman ps -a --format '{{.Names}} {{.Status}}' # includes dead ones
ls ~/.config/containers/systemd/*.container # what should be boot-persistent
systemctl --user is-enabled <name>.service # expect: generated
```
A container in `ps -a` that is `Exited(…)` **and has no unit** is the classic
case: rootless podman does not restart bare containers after the stop that
shutdown performs, so a compose-only service dies at the first reboot and
stays dead. If the compose file still names it, the repo's own rule is being
violated — fix by writing the quadlet (mirror the same service on the other
host), deleting the compose entry (one owner per container name), and making
the deploy path reach it: planner service-set + path rule, dispatcher
allowlist `<target>:<service>`, handler installing from `"$TARGET/..."` never
a hardcoded peer's path.
**2. Classify every down target — do not report a count.**
`targets?state=active` gives health **and** last error; two down targets can
mean opposite things:
| Symptom | Meaning |
|---|---|
| `connection refused` | service not listening → real outage (podman ps, logs, exit codes) |
| `404` on `/metrics` | exporter path wrong or metrics disabled → config defect, service fine |
| `unsupported Content-Type …` | scraping an HTML *page* as metrics → probe inert; Prometheus 3 refuses HTML; a real probe needs a blackbox exporter |
| target absent | §1 |
```bash
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | python3 -c "
import json,sys
for t in json.load(sys.stdin)['data']['activeTargets']:
if t['health'] != 'up':
print(t['labels']['job'], t['scrapeUrl'], (t['lastError'] or '')[:80])"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' http://<target>/metrics # hand-probe before believing the scraper
```
"7 of 11 up" without the four reasons is how a real outage hid behind three
config defects on this fleet — the crash-looping GPU service was one of four
failures reported as a number.
**3. Exposure posture while you are in there.** An exporter bound to the
wildcard publishes an unauthenticated API on every interface. Expect exactly
two listeners per host-exporter — loopback for host-local probes, the host's
own LAN address for the Prometheus container: `ss -ltn | grep <port>` must
show two explicit addresses, never `0.0.0.0`.
**4. Leave decisions that are not yours.** Fix mechanical drift (missing
unit, wrong path, wildcard bind). Do NOT disable a probe to green a
dashboard, or "fix" a crash-looping service whose posture depends on owner
intent — capture the evidence (restart count, exit code, last error line) and
hand it over intact.
## vhost proxy health probes
After deploying or auditing a Caddy/Anubis front door, verify **from a
client**, not from the backend's localhost — that is where transport
mismatches, auth walls, and dead upstreams show up.
```
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/ping
curl -s -o /dev/null -m 6 -w "%{http_code}\n" https://<vhost>/
```
A status code is a diagnosis, not a failure:
- `200` on `/ping` from *arrs = alive; **auth state unknown** (`/ping`
bypasses auth by design) — probe `/` to decide.
- `404` = backend **alive**, route missing (qBittorrent has no `/ping`; 404
there = healthy).
- `502` = Caddy could not reach/negotiate with the upstream: dead backend
**or transport mismatch**. Probe the backend port directly from the proxy's
network (`ssh <proxy-host> curl http://<backend>:<port>/…`). Direct 200 +
vhost 502 ⇒ config bug, not an outage.
- Login page to anonymous (grep body for `password`) = auth ON; instant
redirect to `/transferlist/`-style routes = auth OFF = open door.
**Transport-mismatch mechanism:** `transport http {versions h2c 1.1}` makes
Caddy send an h2c upgrade preface; .NET/Kestrel backends (Jellyfin) reject it
→ 502 per request while the same port answers 200 on plain HTTP/1.1. Fix =
drop the override (default HTTP/1.1). Opposite twin: Plex PMS drops plain
HTTP off-loopback and needs a TLS transport. Match the transport to what the
backend speaks.
**Shell trap when gating commits on a test suite:** `grep -c "^not ok"` exits
**1 when zero mismatches** — inside an `&&` chain it silently kills
everything after it while still printing `0`. Wrap:
`(./tests/run.sh | grep -cE '^not ok' || true) && git commit …`.
## Security posture audit (behavioral)
Answers "what can the internet touch, and is everything else actually
gated?" — **all findings must be behavioral probes**, not config reads.
**1. Inventory (source of truth).** `dankclaw/Caddyfile`: classify every
vhost — guarded (`reverse_proxy anubis:8923`), exempt (direct upstream), or
redirect-only. Public DNS: `dig +short +time=3 +tries=1 <name>.franksdank.cloud
@1.1.1.1` per name — a record at the WAN IP = public; NXDOMAIN = LAN-only by
DNS absence.
**2. Guard chain (expect the challenge, not the app).** Per guarded vhost:
`curl -sk https://<vhost>/` and grep the body for
`anubis|challenge|proof-of-work` (~8 markers on a live challenge page). A 200
with NO markers means the app is served directly — the guard is bypassed.
**3. Exempt vhosts — per-app gating probes.** OpenWebUI `/api/v1/users/me` →
401. Gitea `/api/v1/user` → 401; repo visibility `/api/v1/repos/dank/<repo>`
→ 404 = private. Pi-hole any path → 401 (Caddy basicauth edge). Plex `/web/`
→ 302 (login is JS-rendered). Calibre-Web `/` → 302 to `/login`. Jellyfin
`/Users` → 401. Prowlarr/Sonarr/Radarr `/api/v3/system/status`; Lidarr/Readarr
`/api/v1/system/status` → 401 = gated, 200 = fully open. qBittorrent
`/api/v2/torrents/info` → 403 without SID. Hermes/hermesadmin `/` → 302 to
`/login`. IdP (`auth.`): `/` → 302 → `/if/flow/default-authentication-flow/`
→ 200 (flow HTML is React; grep "password" finds nothing — the 200 on the
flow URL is the proof).
**4. WAN port scan — from an EXTERNAL vantage.** LAN→WAN hairpin is
unreliable; scan from gluetun's ProtonVPN egress:
```bash
ssh dankclaw 'podman exec gluetun sh -c "for p in 21 22 53 80 443 3000 3005 8080 \
8081 8082 8083 8085 8096 8787 8923 8989 9000 9090 9119 9696 7878 8686 3100 \
5055 32400 7770 7771 2019; do timeout 2 wget -qO- http://71.227.227.172:\$p/ \
>/dev/null 2>&1 && echo OPEN: \$p; done"'
```
Confirm the egress is truly external first (`wget -qO- https://api.ipify.org`
→ ProtonVPN IP, not Comcast). **Trap: plain-HTTP probes against TLS ports
(443, 32400) are false negatives** — port open, protocol mismatch; re-probe
with `wget -qO- https://IP[:port]/`. busybox wget has no `--secure` flag; it
errors instantly and the `&&` chain silently skips the echo (a sub-second
wall time on a "timeout" scan means the flag failed, not that ports closed).
**5. Residuals.** Grafana basic-auth residual: `curl -u admin:admin
http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
auth (OpenAI API, no token). Stale-record check: current WAN IP =
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
compare against every public A record — a record at a different Comcast IP
is stale.
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
(dead config, no exposure).
## Stash supersession audit before drop
Use when a repo (especially a live-deployment checkout where the working tree
IS the deployment) has an old stash whose content may have been superseded by
later commits, and dropping blindly risks losing unique facts.
1. **Inventory**: `git stash show --stat stash@{0}`.
2. **Probe for unique content per file**: `git show "stash@{0}:<path>"` and
diff against HEAD; count lines present only in the stash version. A file
with zero unique-vs-HEAD lines is fully superseded — safe to drop.
3. **Classify each unique-content hit**: *stale fact* (superseded by a later
refactor, e.g. old model naming) → drop is correct; *live durable fact*
(an operational invariant never re-recorded) → extract it into live
docs/memory or flag to the user before dropping.
4. **Archive before dropping** — zero loss: tag the stash commit, then drop:
```bash
git tag archive/stash-<YYYY-MM-DD>-<base-branch>-wip "stash@{0}"
git stash drop "stash@{0}"
```
The tag pins the full stash object including its index parent; recoverable
via `git show archive/stash-…` or `git checkout archive/stash-…^3 -- <path>`
for the untracked-files third parent. Stashes are invisible in normal history
and expire from reflogs (~90 days); tagging converts "drop" into a
reversible archive at zero workflow cost.
## Whole-fleet audit (git/test/live/docs cross-check)
Deep-audit the homelab fleet — git/test state, live service health, and
docs-vs-live cross-check. Run from the dev checkout
(`/home/blank/Projects/serverAppConfig`).
**1. Probe the live fleet — self-identifying, ONE ssh per host.** Do NOT
trust the display order of parallel SSH outputs: they can render crossed and
you will attribute host A's output to host B (2026-09-21: a false host↔IP
"conflict" dissolved by a single self-contained per-host probe). Put
`hostname` + a distinctive marker + the LLM identity in the SAME command:
```
for ip in 192.168.0.143 192.168.0.235; do
ssh -o BatchMode=yes dank@$ip \
'echo "host=$(hostname)"; \
echo "romm=$(podman ps -q --filter name=^romm$|wc -l) caddy=$(podman ps -q --filter name=^caddy$)"; \
echo -n "llm7770="; curl -s -m3 http://127.0.0.1:7770/v1/models|head -c60; echo'
done
```
Ground truth (verify, don't assume — IPs can move): 192.168.0.143 = dankclaw
= AI stack + `frank-large` :7770 + `frank-small` :7771 (RTX 3090 + Strix
Halo iGPU); 192.168.0.235 = dankserver = media stack + `frank-primary`
:7770 (2× R9700). Repo dirs + live hostnames agree; the deploy tooling is
hostname-anchored (`ssh dankclaw`/`ssh dankserver` via `~/.ssh/config`), so
it is unaffected by IP drift.
**2. LLM endpoints — health + identity.** `curl /health` per endpoint, then
`/v1/models` (first 120 chars) — a 200 with the right model `id`/`name` is
the pass condition.
**3. Tests — the deploy completeness gate.** `./tests/run.sh` (bats) must
end `ALL TESTS PASSED`; `tests/e2e/run.sh` runs Playwright against the LIVE
fleet. **A single-test flake is normal**, not a regression: the plex crawl
intermittently lands on `app.plex.tv`, the grafana SSO test occasionally
times out, and the FrankBook-Pro leg fails when the mini sleeps. Re-run just
that spec (`npx playwright test --project=standard specs/<n>-<name>.spec.ts`)
to classify before alarming. A `deploy.sh` e2e gate can fail on a flake
while the change itself is fine — verify the actual change live, then re-run
the suite.
**4. Repo state.** `git status --short --branch; git log --oneline -20;
git stash list; git worktree list` (prune stale worktrees), and
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
**shared checkout with parallel instances**: uncommitted peer WIP and
stash-list churn are expected. Never commit or drop a peer's files. To
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
**5. Prometheus — expected-down targets (not outages).** Query
`/api/v1?query=up` from inside the container; the structurally red-by-
construction jobs (`comfyui` HTML scrape, `services` up/down-as-scrape,
`gitea` exporter off) are documented in [[Observability]] — anything else
down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
a *host bind*, not a published port, so the bridge-networked Prometheus
container cannot reach it at the host's LAN IP — netavark DNATs published
ports only. Scrape same-host services via
`host.containers.internal:<port>`; verify the fix from inside the container
before editing: `podman exec prometheus wget -qO-
http://host.containers.internal:<port>/metrics`.
**6. Cross-check docs vs live.** Confirm the AGENTS.md host↔IP↔hardware↔
stack mapping against step 1. The LLM endpoint table and `models.yml` are
probed live and stay current; the prose host labels are the part that goes
stale after an IP move. Report discrepancies; do not "fix" prose from a
single crossed probe.
## Bulk-redact an identity from a docs + config tree
Removing a retired host / machine / project name from a repo that holds both
documentation and live config. Evidence-first, surgical, reversible.
**1. Inventory before touching anything.** Count hits **per file** (the
count decides whether this is a 5-minute job or a user-input decision),
excluding logs/`*.db`/runtime state, and capture `git status --short` first
— you must know which files carry in-flight peer WIP so you never stage
them.
**2. Classify every hit:**
| Bucket | Test | Action |
|---|---|---|
| **Narrative** | prose describing the thing | Reword to the class ("the workstation", "a client box") |
| **Load-bearing** | the identifier IS a lookup key, hostname match, or config value | **Keep.** Prove it: read the code that consumes it |
| **Evidence** | quoted logs, transcripts, reports | Keep timestamps/lines; de-identify only the host field |
Load-bearing examples that look like prose but are not: `install.sh` doing
`PKGS=("common.pkgs" "$HOST.pkgs")` with `HOST="$(hostname)"` (the manifest
*filename* is the key); a `--host <name>` case list; a Wi-Fi SSID. Hardware
model strings, driver versions, PCI addresses are subject matter, not
identity. State the boundary in the commit message.
**3. Subdirectories: distill, don't rename or keep.** For a per-host doc
tree (`docs/system/<host>/`), ask before acting — rename/trash/distill are
genuinely different. When distilling: read every page, sort into
still-true-and-machine-independent (fold into a general page) vs
specific-to-the-retired-box (let go), then `mv` the directory to
`.AGENTTRASH/YYYYMMDD-topic/` and repoint every index. Wikilinks resolve by
filename anywhere in the tree, so deleting a page can break a link in a
file you never grepped — run the link checker.
**4. Execute with count assertions, never blind sed.** Per-file mapping of
`(path, old_exact_string, new_string, expected_count)`; abort the entry if
the count differs; exact strings, not regex. **Do not hand this to a
subagent** — observed: dropped list markers so bullets stopped rendering, a
numbered list with two `3.` items, "corrected" paths to nonexistent
directories, a deleted sentence where a reword was required. If a subagent
already did it: `git checkout --` its touched paths (excluding the WIP
files from step 1) and redo it.
**5. Audit:** `grep -rniE '<name>|<its-ip>' <scope>` (expect nothing), then
`git diff -U0 -- <scope> | grep -E '^\+[^+]'` and **read every added line**
— the diff is the step that catches damage; the grep only proves a string
is gone. Then the repo's own linter.
**6. Commit scoped** — only your own paths, never `git add -A`. If the
checkout IS the live deployment, committing on the current branch is usually
correct; a branch switch can swap live config out from under a running
process. Push/PR only when asked.
**7. Report the residue honestly** — what still carries the name and why:
excluded scopes, load-bearing config, git history, and the cost of rewriting
history (force-push + re-clones). "Purged from the working tree" is a true
claim; "purged" alone is not.
## Related
- [[Observability]] — the monitored stack
- [[Stale Signal Audit]] — the down-target decision table
- [[Observability Provisioning]] — adding scrape targets and rules
- [[Homepage Status Probes]] — the status page that renders probe states
- [[Deploy State Triage]] — the deploy plumbing the audits exercise