serverAppConfigs/serverAppConfigs-docs/Architecture/Stale Signal Audit.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Stale Signal Audit	The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.	testing observability audit signals gates	2026-10-05T00:00:00-07:00

Stale Signal Audit

Run after any change and before trusting a green gate. The productive question is which signal here cannot possibly be correct? Every pass on this repo (4 of 4) found real defects — including one in the rule file every future agent reads first.

The surfaces, and what each vantage point can reach

Surface	Producer's vantage point	Can reach
Homepage status cards	the homepage-status timer on Dankclaw	Dankclaw's loopback, Dankclaw container names, published LAN addresses
Prometheus scrape jobs	the prometheus container on Dankclaw	container names on dankclaw_default, LAN addresses, that host's loopback only
bats / e2e suites	the dev box	the repo, plus the live fleet over HTTPS/LAN

Anything else — another host's loopback, a container-internal port never published — is unreachable no matter how healthy the service is. The failure mode is always the same shape: a signal that cannot succeed by construction, which teaches the operator to ignore the surface it lives on.

1. Permanently-red probes (reachability)

For every monitor/probe/alert target ask: can this checker physically reach that endpoint? Seen classes: a card probing another host's container-internal port; a "probe" job scraping / (HTML → unsupported Content-Type); a /metrics that is disabled (Gitea 404).

• A scrape cannot express "is this HTTP service answering?" — the right probe is the blackbox exporter in probe mode (which also gives guarded vhosts a real uptime signal, since Prometheus cannot solve an Anubis challenge).
• Fix in the direction of honesty, never of silence: unreachable target → mark the homepage card wip: true with the reason in description (the checker skips wip; wip is the existing convention for "real service, not probeable"), or repoint the job at something reachable. NEVER open a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable.
• Alert rules must not depend on always-red signals — scope them to services that can actually be healthy (LlmEndpointDown is scoped to frank-* for exactly that reason).
• A target's failure STRING can prove a fix: a /metrics scrape flipping from connection refused to unsupported Content-Type "text/html" means the service came up and only the scrape is wrong. Read the error, not just up/down.

2. Vacuous assertions (highest severity — the gate is already dead)

A seam that greps a file passes by absence once the file is retired: grep exits 2 on a missing path and ! inverts it.

grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"

Fix shape — one place asserts the invariant, each local site keeps a live check:

@test "no <host> compose file competes with the quadlet units" {
    local cf="$REPO_ROOT/<host>/compose.yaml"
    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"

Prove teeth: recreate the offending file (printf 'services:\n <svc>:\n image: x\n' > <path>), run the seams, confirm RED, then move the file to trash. A re-armed check you have never seen fail is still unverified.

3. Gates that never run

Playwright specs in tests/e2e/ are matched by number prefix in playwright.config.ts; an unregistered spec silently never runs and the suite count stays the same. Count tests before/after; tests/e2e_scaffold.bats fails on an unregistered spec by design.

4. Validate the checker before acting on it

A checker's output is a hypothesis about another tool's semantics. Modelling Obsidian with naive text matching produced 45 "broken links" of which 3 were real — the rest were case-sensitivity (a qBittorrent link to qbittorrent.md), path-qualified resolution (a path-qualified folder/Note link resolves by path SUFFIX, not stem), and %20 URL-encoding in markdown links. Fix the checker first, re-derive the report, only then edit documents. The result lives in scripts/check-vault-links.py with those three rules in its docstring, gated by the no unresolved links in the docs vault seam. Same class: a find … | head -1 selector that reads the wrong file; a truncated listing that makes you "fix" a file that exists.

5. Prose invalidated by your own change

After retiring a file or adding a feature, grep for EVERY reference, not just code:

grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .

This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long after it was restored with GPU access; a unit said "there are no alerting rules" hours after five were added; a docs page documented three scrape jobs as working when they could never report up. Clean cutover includes prose — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale).

Order and verification

1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates, docs/rule files, dashboard panels).
2. For each, ask what the signal CAN express, not what it says.
3. Validate any checker before editing the things it flags.
4. Fix at the root; never weaken an assertion to make it pass.
5. Gate the class so it cannot return (a seam), then prove the seam fails on the defect.

Verify: ./scripts/check-vault-links.py (exit 1 on breaks); query each Prometheus panel expression against the API (0 series on a panel that should show data is a defect; 0 ALERTS when nothing fires is not); read the deployed /srv/homepage/status.json and triage every state != up entry against reality; query /api/v1/targets?state=active and print health != "up" with lastError; ./tests/run.sh → ALL TESTS PASSED. Report per finding: the surface, why the signal could not have been correct, the fix, and the red/green evidence — flag any regression you introduced yourself in the same breath (one was found this way: a tightened bind killed a reverse-proxied vhost while every gate stayed green, because the spec covering that hostname did not exist).

Related

• Testing Policy — the seam inventory this audit walks
• Observability — the scrape/probe surfaces it audits
• Quadlet Lifecycle — the ownership labels the probes read