---
name: status-surface-dead-signal-audit
description: "Audit a Dank-fleet status surface (homepage cards, Prometheus targets, tests) for entries that can never be green because the producer cannot reach the target, classify them as outage vs config defect, and fix honestly (wip, repoint, blackbox) instead of silencing them."
---

# Dead-signal audit on the Dank fleet's status surfaces

Use when a status board, Prometheus target list, alert set or test suite has
permanently non-green entries, or before trusting one of those surfaces. The
failure mode is always the same shape: a signal that **cannot** succeed by
construction, which teaches the operator to ignore the surface it lives on.

## The surfaces, and what can actually be probed

| Surface | Producer's vantage point | Can reach |
|---|---|---|
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, loopback of that host only |
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |

Anything else — another host's loopback, or a container-internal port that was
never published — is unreachable no matter how healthy the service is.

## Procedure

1. **List the non-green entries.** Homepage: `python3 - boardcheck.py
   /srv/homepage/status.json` on Dankclaw (`status.json` holds one dict per card
   with `state` up/down/wip). Prometheus: query
   `/api/v1/targets?state=active` and print `health != "up"` with `lastError`.
2. **Classify each one by producer reach**, not by the service. Ask: can the
   thing that probes it get there at all?
3. **Fix in the direction of honesty, never of silence:**
   - unreachable target → mark the card `wip` with the reason in its
     `description` (the checker skips `wip`), or repoint the job at something
     reachable. Do **not** open a database port on the LAN to make a badge
     green, and do **not** delete a scrape job — an absent scrape and a healthy
     one are indistinguishable.
   - wrong path / wrong content type → fix the path (e.g. Gitea's `/metrics`
     needs metrics enabled), or move page-only targets to a blackbox exporter in
     probe mode.
   - failing test → fix the selector. A chronic red test is a defect with an
     owner, not a baseline (`find … | head -1` picked another service's file).
4. **Verify on the surface itself**, not in the config: re-read `status.json`
   (expect 0 `down`), query `up == 0` targets, run the suite to green.
5. Write the rule where the next person will read it — the Homepage and
   Observability pages both state it now.

## Traps

- The producer's vantage point is the whole question; a card or job can be
  "down" for a service that answers perfectly from the host it runs on.
- `wip` is the existing convention for "real service, not probeable" (Gluetun,
  WireGuard/UDP), not a suppression: keep the reason in the description.
- Alert rules must not depend on signals that are always red; scope them to
  the services that can actually be healthy (`LlmEndpointDown` is scoped to
  `frank-*` for exactly that reason).