--- type: reference title: Stale Signal Audit description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time. tags: [testing, observability, audit, signals, gates] timestamp: 2026-10-05T00:00:00-07:00 --- # Stale Signal Audit Run after any change and before trusting a green gate. The productive question is **which signal here cannot possibly be correct?** Every pass on this repo (4 of 4) found real defects — including one in the rule file every future agent reads first. ## The surfaces, and what each vantage point can reach | Surface | Producer's vantage point | Can reach | |---|---|---| | Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses | | Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only | | bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN | Anything else — another host's loopback, a container-internal port never published — is unreachable no matter how healthy the service is. The failure mode is always the same shape: a signal that **cannot** succeed by construction, which teaches the operator to ignore the surface it lives on. ## 1. Permanently-red probes (reachability) For every monitor/probe/alert target ask: *can this checker physically reach that endpoint?* Seen classes: a card probing another host's container-internal port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a `/metrics` that is disabled (Gitea 404). - A scrape cannot express "is this HTTP service answering?" — the right probe is the **blackbox exporter** in probe mode (which also gives guarded vhosts a real uptime signal, since Prometheus cannot solve an Anubis challenge). - **Fix in the direction of honesty, never of silence:** unreachable target → mark the homepage card `wip: true` with the reason in `description` (the checker skips `wip`; `wip` is the existing convention for "real service, not probeable"), or repoint the job at something reachable. NEVER open a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable. - Alert rules must not depend on always-red signals — scope them to services that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for exactly that reason). - A target's failure STRING can prove a fix: a `/metrics` scrape flipping from `connection refused` to `unsupported Content-Type "text/html"` means the service came up and only the scrape is wrong. Read the error, not just up/down. ## 2. Vacuous assertions (highest severity — the gate is already dead) A seam that greps a file passes *by absence* once the file is retired: `grep` exits 2 on a missing path and `!` inverts it. ```bash grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50 [ -f "$path" ] || echo "MISSING: a seam may be grepping this" ``` Fix shape — one place asserts the invariant, each local site keeps a live check: ```bats @test "no compose file competes with the quadlet units" { local cf="$REPO_ROOT//compose.yaml" [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; } } # at the per-service site: still fails if the entry returns [ ! -f "$cf" ] || ! grep -qE '^\s+:' "$cf" ``` **Prove teeth:** recreate the offending file (`printf 'services:\n :\n image: x\n' > `), run the seams, confirm RED, then move the file to trash. A re-armed check you have never seen fail is still unverified. ## 3. Gates that never run Playwright specs in `tests/e2e/` are matched by **number prefix** in `playwright.config.ts`; an unregistered spec silently never runs and the suite count stays the same. Count tests before/after; `tests/e2e_scaffold.bats` fails on an unregistered spec by design. ## 4. Validate the checker before acting on it A checker's output is a hypothesis about another tool's semantics. Modelling Obsidian with naive text matching produced 45 "broken links" of which 3 were real — the rest were case-sensitivity (a `qBittorrent` link to `qbittorrent.md`), path-qualified resolution (a path-qualified `folder/Note` link resolves by path SUFFIX, not stem), and `%20` URL-encoding in markdown links. Fix the checker first, re-derive the report, only then edit documents. The result lives in `scripts/check-vault-links.py` with those three rules in its docstring, gated by the `no unresolved links in the docs vault` seam. Same class: a `find … | head -1` selector that reads the wrong file; a truncated listing that makes you "fix" a file that exists. ## 5. Prose invalidated by your own change After retiring a file or adding a feature, grep for EVERY reference, not just code: ```bash grep -rn "" --include="*" . | grep -v '^./.git/' grep -rn "no \|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" . ``` This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long after it was restored with GPU access; a unit said "there are no alerting rules" hours after five were added; a docs page documented three scrape jobs as working when they could never report up. **Clean cutover includes prose** — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale). ## Order and verification 1. Enumerate the surfaces (status board, metrics targets, alert rules, test gates, docs/rule files, dashboard panels). 2. For each, ask what the signal CAN express, not what it says. 3. Validate any checker before editing the things it flags. 4. Fix at the root; never weaken an assertion to make it pass. 5. Gate the class so it cannot return (a seam), then prove the seam fails on the defect. Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each Prometheus panel expression against the API (0 series on a panel that should show data is a defect; 0 `ALERTS` when nothing fires is not); read the deployed `/srv/homepage/status.json` and triage every `state != up` entry against reality; query `/api/v1/targets?state=active` and print `health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`. Report per finding: the surface, why the signal could not have been correct, the fix, and the red/green evidence — flag any regression you introduced yourself in the same breath (one was found this way: a tightened bind killed a reverse-proxied vhost while every gate stayed green, because the spec covering that hostname did not exist). ## Related - [[Testing Policy]] — the seam inventory this audit walks - [[Observability]] — the scrape/probe surfaces it audits - [[Quadlet Lifecycle]] — the ownership labels the probes read