Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
146 lines
6.8 KiB
Markdown
146 lines
6.8 KiB
Markdown
---
|
|
type: reference
|
|
title: Stale Signal Audit
|
|
description: "The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run."
|
|
tags: [testing, observability, audit, signals, gates]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Stale Signal Audit
|
|
|
|
Run after any change and before trusting a green gate. The productive question
|
|
is **which signal here cannot possibly be correct?** Every pass on this repo
|
|
(4 of 4) found real defects — including one in the rule file every future
|
|
agent reads first.
|
|
|
|
## The surfaces, and what each vantage point can reach
|
|
|
|
| Surface | Producer's vantage point | Can reach |
|
|
|---|---|---|
|
|
| Homepage status cards | the `homepage-status` timer on **Dankclaw** | Dankclaw's loopback, Dankclaw container names, **published** LAN addresses |
|
|
| Prometheus scrape jobs | the `prometheus` container on Dankclaw | container names on `dankclaw_default`, LAN addresses, that host's loopback only |
|
|
| bats / e2e suites | the dev box | the repo, plus the live fleet over HTTPS/LAN |
|
|
|
|
Anything else — another host's loopback, a container-internal port never
|
|
published — is unreachable no matter how healthy the service is. The failure
|
|
mode is always the same shape: a signal that **cannot** succeed by
|
|
construction, which teaches the operator to ignore the surface it lives on.
|
|
|
|
## 1. Permanently-red probes (reachability)
|
|
|
|
For every monitor/probe/alert target ask: *can this checker physically reach
|
|
that endpoint?* Seen classes: a card probing another host's container-internal
|
|
port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
|
`/metrics` that is disabled (Gitea 404).
|
|
|
|
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
|
is the **blackbox exporter** in probe mode
|
|
(`/probe?target=…&module=http_2xx`), which also gives guarded vhosts
|
|
a real uptime signal, since Prometheus cannot solve an Anubis challenge.
|
|
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
|
mark the homepage card `wip: true` with the reason in `description` (the
|
|
checker skips `wip`; `wip` is the existing convention for "real service,
|
|
not probeable"), or repoint the job at something reachable. NEVER open a
|
|
database port on the LAN to make a badge green, and never delete a scrape
|
|
job — an absent scrape and a healthy one are indistinguishable.
|
|
- Alert rules must not depend on always-red signals — scope them to services
|
|
that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries
|
|
is recorded on [[Prometheus]].
|
|
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
|
`connection refused` to `unsupported Content-Type "text/html"` means the
|
|
service came up and only the scrape is wrong. Read the error, not just
|
|
up/down.
|
|
|
|
## 2. Vacuous assertions (highest severity — the gate is already dead)
|
|
|
|
A seam that greps a file passes *by absence* once the file is retired:
|
|
`grep` exits 2 on a missing path and `!` inverts it.
|
|
|
|
```bash
|
|
grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
|
|
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"
|
|
```
|
|
|
|
Fix shape — one place asserts the invariant, each local site keeps a live
|
|
check:
|
|
|
|
```bats
|
|
@test "no <host> compose file competes with the quadlet units" {
|
|
local cf="$REPO_ROOT/<host>/compose.yaml"
|
|
[ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
|
|
}
|
|
# at the per-service site: still fails if the entry returns
|
|
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"
|
|
```
|
|
|
|
**Prove teeth:** recreate the offending file
|
|
(`printf 'services:\n <svc>:\n image: x\n' > <path>`), run the seams,
|
|
confirm RED, then move the file to trash. A re-armed check you have never
|
|
seen fail is still unverified.
|
|
|
|
## 3. Gates that never run
|
|
|
|
Playwright specs in `tests/e2e/` are matched by **number prefix** in
|
|
`playwright.config.ts`; an unregistered spec silently never runs and the
|
|
suite count stays the same. Count tests before/after;
|
|
`tests/e2e_scaffold.bats` fails on an unregistered spec by design.
|
|
|
|
## 4. Validate the checker before acting on it
|
|
|
|
A checker's output is a hypothesis about another tool's semantics. Modelling
|
|
Obsidian with naive text matching produced 45 "broken links" of which 3 were
|
|
real — the rest were case-sensitivity (a `qBittorrent` link to
|
|
`qbittorrent.md`), path-qualified resolution (a path-qualified
|
|
`folder/Note` link resolves by path SUFFIX, not
|
|
stem), and `%20` URL-encoding in markdown links. Fix the checker first,
|
|
re-derive the report, only then edit documents. The result lives in
|
|
`scripts/check-vault-links.py` with those three rules in its docstring,
|
|
gated by the `no unresolved links in the docs vault` seam. Same class: a
|
|
`find … | head -1` selector that reads the wrong file; a truncated listing
|
|
that makes you "fix" a file that exists.
|
|
|
|
## 5. Prose invalidated by your own change
|
|
|
|
After retiring a file or adding a feature, grep for EVERY reference, not just
|
|
code:
|
|
|
|
```bash
|
|
grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
|
|
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .
|
|
```
|
|
|
|
This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long
|
|
after it was restored with GPU access; a unit said "there are no alerting
|
|
rules" hours after five were added; a docs page documented three scrape jobs
|
|
as working when they could never report up. **Clean cutover includes prose** —
|
|
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
|
be stale).
|
|
|
|
## Order and verification
|
|
|
|
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
|
gates, docs/rule files, dashboard panels).
|
|
2. For each, ask what the signal CAN express, not what it says.
|
|
3. Validate any checker before editing the things it flags.
|
|
4. Fix at the root; never weaken an assertion to make it pass.
|
|
5. Gate the class so it cannot return (a seam), then prove the seam fails on
|
|
the defect.
|
|
|
|
Verify: `./scripts/check-vault-links.py` (exit 1 on breaks); query each
|
|
Prometheus panel expression against the API (0 series on a panel that should
|
|
show data is a defect; 0 `ALERTS` when nothing fires is not); read the
|
|
deployed `/srv/homepage/status.json` and triage every `state != up` entry
|
|
against reality; query `/api/v1/targets?state=active` and print
|
|
`health != "up"` with `lastError`; `./tests/run.sh` → `ALL TESTS PASSED`.
|
|
Report per finding: the surface, why the signal could not have been correct,
|
|
the fix, and the red/green evidence — flag any regression you introduced
|
|
yourself in the same breath (one was found this way: a tightened bind killed
|
|
a reverse-proxied vhost while every gate stayed green, because the spec
|
|
covering that hostname did not exist).
|
|
|
|
## Related
|
|
|
|
- [[Testing Policy]] — the seam inventory this audit walks
|
|
- [[Observability]] — the scrape/probe surfaces it audits
|
|
- [[Quadlet Lifecycle]] — the ownership labels the probes read
|