Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
57 lines
3.2 KiB
Markdown
57 lines
3.2 KiB
Markdown
---
|
|
name: status-surface-dead-signal-audit
|
|
description: "Audit a Dank-fleet status surface (homepage cards, Prometheus targets, tests) for entries that can never be green because the producer cannot reach the target, classify them as outage vs config defect, and fix honestly (wip, repoint, blackbox) instead of silencing them."
|
|
---
|
|
|
|
# Dead-signal audit on the Dank fleet's status surfaces
|
|
|
|
Use when a status board, Prometheus target list, alert set or test suite has
|
|
permanently non-green entries, or before trusting one of those surfaces. The
|
|
failure mode is always the same shape: a signal that **cannot** succeed by
|
|
construction, which teaches the operator to ignore the surface it lives on.
|
|
|
|
## The surfaces, and what can actually be probed
|
|
|
|
| Surface | Producer's vantage point | Can reach |
|
|
|---|---|---|
|
|
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
|
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, loopback of that host only |
|
|
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
|
|
|
Anything else — another host's loopback, or a container-internal port that was
|
|
never published — is unreachable no matter how healthy the service is.
|
|
|
|
## Procedure
|
|
|
|
1. **List the non-green entries.** Homepage: `python3 - boardcheck.py
|
|
/srv/homepage/status.json` on Dankclaw (`status.json` holds one dict per card
|
|
with `state` up/down/wip). Prometheus: query
|
|
`/api/v1/targets?state=active` and print `health != "up"` with `lastError`.
|
|
2. **Classify each one by producer reach**, not by the service. Ask: can the
|
|
thing that probes it get there at all?
|
|
3. **Fix in the direction of honesty, never of silence:**
|
|
- unreachable target → mark the card `wip` with the reason in its
|
|
`description` (the checker skips `wip`), or repoint the job at something
|
|
reachable. Do **not** open a database port on the LAN to make a badge
|
|
green, and do **not** delete a scrape job — an absent scrape and a healthy
|
|
one are indistinguishable.
|
|
- wrong path / wrong content type → fix the path (e.g. Gitea's `/metrics`
|
|
needs metrics enabled), or move page-only targets to a blackbox exporter in
|
|
probe mode.
|
|
- failing test → fix the selector. A chronic red test is a defect with an
|
|
owner, not a baseline (`find … | head -1` picked another service's file).
|
|
4. **Verify on the surface itself**, not in the config: re-read `status.json`
|
|
(expect 0 `down`), query `up == 0` targets, run the suite to green.
|
|
5. Write the rule where the next person will read it — the Homepage and
|
|
Observability pages both state it now.
|
|
|
|
## Traps
|
|
|
|
- The producer's vantage point is the whole question; a card or job can be
|
|
"down" for a service that answers perfectly from the host it runs on.
|
|
- `wip` is the existing convention for "real service, not probeable" (Gluetun,
|
|
WireGuard/UDP), not a suppression: keep the reason in the description.
|
|
- Alert rules must not depend on signals that are always red; scope them to
|
|
the services that can actually be healthy (`LlmEndpointDown` is scoped to
|
|
`frank-*` for exactly that reason).
|