Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
72 lines
4.1 KiB
Markdown
72 lines
4.1 KiB
Markdown
---
|
|
name: stale-signal-audit
|
|
description: "Audit homelab/fleet surfaces (status boards, metrics targets, test gates, docs, rule files) for signals that are already wrong — impossible-to-pass probes, assertions that pass by absence, linters that produce false positives, and prose invalidated by your own change. Use after a change lands, when a gate looks green but you want to know what it actually proves, or when asked what is broken."
|
|
---
|
|
|
|
# Stale-signal audit
|
|
|
|
Signals rot without failing. This is the battery that found real defects on 4 of 4
|
|
passes in serverAppConfig; run it after any change and before trusting a green gate.
|
|
|
|
## The five checks
|
|
|
|
1. **Probe reachability.** For every monitor/probe/alert target, ask: *can this
|
|
checker physically reach that endpoint?* A probe that cannot succeed reports a
|
|
false outage forever, and an always-red signal teaches the operator to ignore
|
|
the surface.
|
|
- Homepage (`shared/homepage/services.json` + `homepage-status` on Dankclaw): the
|
|
monitor sees Dankclaw's loopback, container names, and **published** LAN ports
|
|
only. A card probing another host's container-internal port can never be up →
|
|
mark `wip` with the reason and let the app's card carry the status.
|
|
- Prometheus: a *scrape* cannot express "is this HTTP service answering?" —
|
|
scraping an HTML page gives `unsupported Content-Type`; a `/metrics` that 404s
|
|
stays down. Remedy: blackbox exporter in probe mode. Never "fix" a probe by
|
|
publishing a database or adding a wildcard bind.
|
|
|
|
2. **Assertions that pass by absence.** Grep-based tests against a file that was
|
|
deleted or retired pass vacuously: a missing path makes `grep` exit 2 and a
|
|
negated `! grep` succeeds. Look for tests referencing paths that no longer exist
|
|
(`git ls-files` the target, or just `[ -f ]` it).
|
|
- Fix: assert the invariant directly ("no <file> competes with the unit"), keep a
|
|
conditional live check for the file returning, and drop dead variables that
|
|
only referenced it.
|
|
- **Prove teeth**: recreate the file/condition and watch the test go red. A test
|
|
you have never seen fail is not a test.
|
|
|
|
3. **Linter validity.** Before acting on any checker's output, validate the checker
|
|
against the resolver/contract it models. Obsidian example: wikilinks are
|
|
case-insensitive, `[[folder/Note]]` resolves by **path suffix** not stem, and
|
|
`%20` in markdown links is URL-encoding — a naive checker reported 45 broken
|
|
links when 2 were real. Same class: a `find … | head -1` selector that reads the
|
|
wrong file; a truncated listing that makes you "fix" a file that exists.
|
|
|
|
4. **Gates that never run.** Playwright specs in `tests/e2e/` are matched by
|
|
**number prefix** in `playwright.config.ts`; an unregistered spec silently never
|
|
runs and the suite count stays the same. Verify by counting tests before/after
|
|
(`tests/e2e_scaffold.bats` now fails on an unregistered spec).
|
|
|
|
5. **Prose and comments invalidated by your own change.** Clean cutover includes
|
|
docs, unit docstrings, and the repo's rule file (`AGENTS.md`) — which is what
|
|
every future agent reads first, so a stale claim there is the most expensive kind.
|
|
Check: runbooks/rows naming a retired service, "unconfigured" claims after you
|
|
configured something, upgrade instructions pointing at a deleted file, and
|
|
lessons that the deployment has since contradicted.
|
|
|
|
## Order
|
|
|
|
1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates,
|
|
docs/rule files, dashboard panels).
|
|
2. For each, ask what the signal *can* express, not what it says.
|
|
3. Validate any checker before editing the things it flags.
|
|
4. Fix at the root; never weaken an assertion to make it pass.
|
|
5. Gate the class so it cannot return (a seam), then prove the seam fails on the
|
|
defect.
|
|
|
|
## Verification
|
|
|
|
- Vault links: `./scripts/check-vault-links.py` (exit 1 on breaks).
|
|
- Prometheus panels/rules: query each panel expression against the API — 0 series on
|
|
a panel that should show data is a defect, 0 `ALERTS` when nothing fires is not.
|
|
- Suite: `./tests/run.sh`; declares `ALL TESTS PASSED`.
|
|
- A gate you changed: recreate the defect and confirm red.
|