serverAppConfigs/serverAppConfigs-docs/Architecture/Stale Signal Audit.md

---
type: reference
title: Stale Signal Audit
description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
tags: [testing, observability, audit, signals, gates]
timestamp: 2026-10-05T00:00:00-07:00
---

# Stale Signal Audit

Run after any change and before trusting a green gate. The productive question
is **which signal here cannot possibly be correct?** Every pass on this repo
(4 of 4) found real defects — including one in the rule file every future
agent reads first.

## The surfaces, and what each vantage point can reach

| Surface | Producer's vantage point | Can reach |
|---|---|---|
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |

Anything else — another host's loopback, a container-internal port never
published — is unreachable no matter how healthy the service is. The failure
mode is always the same shape: a signal that **cannot** succeed by
construction, which teaches the operator to ignore the surface it lives on.

## 1. Permanently-red probes (reachability)

For every monitor/probe/alert target ask: *can this checker physically reach
that endpoint?* Seen classes: a card probing another host's container-internal
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
`/metrics` that is disabled (Gitea 404).

- A scrape cannot express "is this HTTP service answering?" — the right probe
  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
- **Fix in the direction of honesty, never of silence:** unreachable target →
  mark the homepage card `wip: true` with the reason in `description` (the
  checker skips `wip`; `wip` is the existing convention for "real service,
  not probeable"), or repoint the job at something reachable. NEVER open a
  database port on the LAN to make a badge green, and never delete a scrape
  job — an absent scrape and a healthy one are indistinguishable.
- Alert rules must not depend on always-red signals — scope them to services
  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
  exactly that reason).
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
  `connection refused` to `unsupported Content-Type "text/html"` means the
  service came up and only the scrape is wrong. Read the error, not just
  up/down.

## 2. Vacuous assertions (highest severity — the gate is already dead)

A seam that greps a file passes *by absence* once the file is retired:
`grep` exits 2 on a missing path and `!` inverts it.

```bash
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
```

Fix shape — one place asserts the invariant, each local site keeps a live
check:

```bats
@test "no <host> compose file competes with the quadlet units" {
    local cf="$REPO_ROOT/<host>/compose.yaml"
    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
```

**Prove teeth:** recreate the offending file
(`printf 'services:\n  <svc>:\n    image: x\n' > <path>`), run the seams,
confirm RED, then move the file to trash. A re-armed check you have never
seen fail is still unverified.

## 3. Gates that never run

Playwright specs in `tests/e2e/` are matched by **number prefix** in
`playwright.config.ts`; an unregistered spec silently never runs and the
suite count stays the same. Count tests before/after;
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.

## 4. Validate the checker before acting on it

A checker's output is a hypothesis about another tool's semantics. Modelling
Obsidian with naive text matching produced 45 "broken links" of which 3 were
real — the rest were case-sensitivity (a `qBittorrent` link to
`qbittorrent.md`), path-qualified resolution (a path-qualified
`folder/Note` link resolves by path SUFFIX, not
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
re-derive the report, only then edit documents. The result lives in
`scripts/check-vault-links.py` with those three rules in its docstring,
gated by the `no unresolved links in the docs vault` seam. Same class: a
`find … | head -1` selector that reads the wrong file; a truncated listing
that makes you "fix" a file that exists.

## 5. Prose invalidated by your own change

After retiring a file or adding a feature, grep for EVERY reference, not just
code:

```bash
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
```

This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
after it was restored with GPU access; a unit said "there are no alerting
rules" hours after five were added; a docs page documented three scrape jobs
as working when they could never report up. **Clean cutover includes prose** —
docs, unit docstrings, and the repo's rule file (the most expensive place to
be stale).

## Order and verification

1. Enumerate the surfaces (status board, metrics targets, alert rules, test
   gates, docs/rule files, dashboard panels).
2. For each, ask what the signal CAN express, not what it says.
3. Validate any checker before editing the things it flags.
4. Fix at the root; never weaken an assertion to make it pass.
5. Gate the class so it cannot return (a seam), then prove the seam fails on
   the defect.

Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
Prometheus panel expression against the API (0 series on a panel that should
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
deployed `/srv/homepage/status.json` and triage every `state != up` entry
against reality; query `/api/v1/targets?state=active` and print
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
Report per finding: the surface, why the signal could not have been correct,
the fix, and the red/green evidence — flag any regression you introduced
yourself in the same breath (one was found this way: a tightened bind killed
a reverse-proxied vhost while every gate stayed green, because the spec
covering that hostname did not exist).

## Related

- [[Testing Policy]] — the seam inventory this audit walks
- [[Observability]] — the scrape/probe surfaces it audits
- [[Quadlet Lifecycle]] — the ownership labels the probes read