serverAppConfigs/serverAppConfigs-docs/Architecture/Stale Signal Audit.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Stale Signal Audit	The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run.	testing observability audit signals gates	2026-10-05T00:00:00-07:00

Stale Signal Audit

Run after any change and before trusting a green gate. The productive question is which signal here cannot possibly be correct? Every pass on this repo (4 of 4) found real defects — including one in the rule file every future agent reads first.

The surfaces, and what each vantage point can reach

Surface	Producer's vantage point	Can reach
Homepage status cards	the homepage-status timer on Dankclaw	Dankclaw's loopback, Dankclaw container names, published LAN addresses
Prometheus scrape jobs	the prometheus container on Dankclaw	container names on dankclaw_default, LAN addresses, that host's loopback only
bats / e2e suites	the dev box	the repo, plus the live fleet over HTTPS/LAN

Anything else — another host's loopback, a container-internal port never published — is unreachable no matter how healthy the service is. The failure mode is always the same shape: a signal that cannot succeed by construction, which teaches the operator to ignore the surface it lives on.

1. Permanently-red probes (reachability)

For every monitor/probe/alert target ask: can this checker physically reach that endpoint? Seen classes: a card probing another host's container-internal port; a "probe" job scraping / (HTML → unsupported Content-Type); a /metrics that is disabled (Gitea 404).

• A scrape cannot express "is this HTTP service answering?" — the right probe is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives guarded vhosts a real uptime signal, since Prometheus cannot solve an Anubis challenge.
• Fix in the direction of honesty, never of silence: unreachable target → mark the homepage card wip: true with the reason in description (the checker skips wip; wip is the existing convention for "real service, not probeable"), or repoint the job at something reachable. NEVER open a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable.
• Alert rules must not depend on always-red signals — scope them to services that can actually be healthy; the frank-* scope LlmEndpointDown carries is recorded on Prometheus.
• A target's failure STRING can prove a fix: a /metrics scrape flipping from connection refused to unsupported Content-Type "text/html" means the service came up and only the scrape is wrong. Read the error, not just up/down.

2. Vacuous assertions (highest severity — the gate is already dead)

A seam that greps a file passes by absence once the file is retired: grep exits 2 on a missing path and ! inverts it.

grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"

Fix shape — one place asserts the invariant, each local site keeps a live check:

@test "no <host> compose file competes with the quadlet units" {
    local cf="$REPO_ROOT/<host>/compose.yaml"
    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"

Prove teeth: recreate the offending file (printf 'services:\n <svc>:\n image: x\n' > <path>), run the seams, confirm RED, then move the file to trash. A re-armed check you have never seen fail is still unverified.

3. Gates that never run

Playwright specs in tests/e2e/ are matched by number prefix in playwright.config.ts; an unregistered spec silently never runs and the suite count stays the same. Count tests before/after; tests/e2e_scaffold.bats fails on an unregistered spec by design.

4. Validate the checker before acting on it

A checker's output is a hypothesis about another tool's semantics. Modelling Obsidian with naive text matching produced 45 "broken links" of which 3 were real — the rest were case-sensitivity (a qBittorrent link to qbittorrent.md), path-qualified resolution (a path-qualified folder/Note link resolves by path SUFFIX, not stem), and %20 URL-encoding in markdown links. Fix the checker first, re-derive the report, only then edit documents. The result lives in scripts/check-vault-links.py with those three rules in its docstring, gated by the no unresolved links in the docs vault seam. Same class: a find … | head -1 selector that reads the wrong file; a truncated listing that makes you "fix" a file that exists.

5. Prose invalidated by your own change

After retiring a file or adding a feature, grep for EVERY reference, not just code:

grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .

This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long after it was restored with GPU access; a unit said "there are no alerting rules" hours after five were added; a docs page documented three scrape jobs as working when they could never report up. Clean cutover includes prose — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale).

Three rules that outlive the incident that taught them

• A target can read down beside a healthy service. Read lastError from /api/v1/targets before believing an outage — llama.cpp refuses /metrics at the protocol layer without --metrics (501) while /health answers 200 for weeks.
• Alert state lives where the tool says it does. Prometheus keeps rule state in process memory — a restart re-stamps activeAt, so a "fresh" alert after a reboot is not evidence of a fresh fault. Alertmanager does persist (--storage.path on a host volume: silences + notification log, Alertmanager).
• Never quote a candidate arm's flags or VRAM as fleet state. A stopped production unit (failed after stop-sigterm timed out. Killing — the known shutdown shape, frank-zealot) beside an answering port means the metric measured a scratch boot; the process shape can change between two censuses ten minutes apart.

Order and verification

1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates, docs/rule files, dashboard panels).
2. For each, ask what the signal CAN express, not what it says.
3. Validate any checker before editing the things it flags.
4. Fix at the root; never weaken an assertion to make it pass.
5. Gate the class so it cannot return (a seam), then prove the seam fails on the defect.

Verify: ./scripts/check-vault-links.py (exit 1 on breaks); query each Prometheus panel expression against the API (0 series on a panel that should show data is a defect; 0 ALERTS when nothing fires is not); read the deployed /srv/homepage/status.json and triage every state != up entry against reality; query /api/v1/targets?state=active and print health != "up" with lastError; ./tests/run.sh → ALL TESTS PASSED. Report per finding: the surface, why the signal could not have been correct, the fix, and the red/green evidence — flag any regression you introduced yourself in the same breath (one was found this way: a tightened bind killed a reverse-proxied vhost while every gate stayed green, because the spec covering that hostname did not exist).

Related

• Testing Policy — the seam inventory this audit walks
• Observability — the scrape/probe surfaces it audits
• Quadlet Lifecycle — the ownership labels the probes read