Files

T

Frank 6bdfd2853e docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.

2026-10-19 00:56:02 -07:00

6.9 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Stale Signal Audit

Run after any change and before trusting a green gate. The productive question is which signal here cannot possibly be correct? Every pass on this repo (4 of 4) found real defects — including one in the rule file every future agent reads first.

The surfaces, and what each vantage point can reach

Surface	Producer's vantage point	Can reach
Homepage status cards	the `homepage-status` timer on Dankclaw	Dankclaw's loopback, Dankclaw container names, published LAN addresses
Prometheus scrape jobs	the `prometheus` container on Dankclaw	container names on `dankclaw_default`, LAN addresses, that host's loopback only
bats / e2e suites	the dev box	the repo, plus the live fleet over HTTPS/LAN

Anything else — another host's loopback, a container-internal port never published — is unreachable no matter how healthy the service is. The failure mode is always the same shape: a signal that cannot succeed by construction, which teaches the operator to ignore the surface it lives on.

1. Permanently-red probes (reachability)

For every monitor/probe/alert target ask: can this checker physically reach that endpoint? Seen classes: a card probing another host's container-internal port; a "probe" job scraping / (HTML → unsupported Content-Type); a /metrics that is disabled (Gitea 404).

A scrape cannot express "is this HTTP service answering?" — the right probe is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives guarded vhosts a real uptime signal, since Prometheus cannot solve an Anubis challenge.
Fix in the direction of honesty, never of silence: unreachable target → mark the homepage card wip: true with the reason in description (the checker skips wip; wip is the existing convention for "real service, not probeable"), or repoint the job at something reachable. NEVER open a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable.
Alert rules must not depend on always-red signals — scope them to services that can actually be healthy; the frank-* scope LlmEndpointDown carries is recorded on Prometheus.
A target's failure STRING can prove a fix: a /metrics scrape flipping from connection refused to unsupported Content-Type "text/html" means the service came up and only the scrape is wrong. Read the error, not just up/down.

2. Vacuous assertions (highest severity — the gate is already dead)

A seam that greps a file passes by absence once the file is retired: grep exits 2 on a missing path and ! inverts it.

grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"

Fix shape — one place asserts the invariant, each local site keeps a live check:

@test "no <host> compose file competes with the quadlet units" {
    local cf="$REPO_ROOT/<host>/compose.yaml"
    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"

Prove teeth: recreate the offending file (printf 'services:\n <svc>:\n image: x\n' > <path>), run the seams, confirm RED, then move the file to trash. A re-armed check you have never seen fail is still unverified.

3. Gates that never run

Playwright specs in tests/e2e/ are matched by number prefix in playwright.config.ts; an unregistered spec silently never runs and the suite count stays the same. Count tests before/after; tests/e2e_scaffold.bats fails on an unregistered spec by design.

4. Validate the checker before acting on it

A checker's output is a hypothesis about another tool's semantics. Modelling Obsidian with naive text matching produced 45 "broken links" of which 3 were real — the rest were case-sensitivity (a qBittorrent link to qbittorrent.md), path-qualified resolution (a path-qualified folder/Note link resolves by path SUFFIX, not stem), and %20 URL-encoding in markdown links. Fix the checker first, re-derive the report, only then edit documents. The result lives in scripts/check-vault-links.py with those three rules in its docstring, gated by the no unresolved links in the docs vault seam. Same class: a find … | head -1 selector that reads the wrong file; a truncated listing that makes you "fix" a file that exists.

5. Prose invalidated by your own change

After retiring a file or adding a feature, grep for EVERY reference, not just code:

grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .

This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long after it was restored with GPU access; a unit said "there are no alerting rules" hours after five were added; a docs page documented three scrape jobs as working when they could never report up. Clean cutover includes prose — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale).

Order and verification

Enumerate the surfaces (status board, metrics targets, alert rules, test gates, docs/rule files, dashboard panels).
For each, ask what the signal CAN express, not what it says.
Validate any checker before editing the things it flags.
Fix at the root; never weaken an assertion to make it pass.
Gate the class so it cannot return (a seam), then prove the seam fails on the defect.

Verify: ./scripts/check-vault-links.py (exit 1 on breaks); query each Prometheus panel expression against the API (0 series on a panel that should show data is a defect; 0 ALERTS when nothing fires is not); read the deployed /srv/homepage/status.json and triage every state != up entry against reality; query /api/v1/targets?state=active and print health != "up" with lastError; ./tests/run.sh → ALL TESTS PASSED. Report per finding: the surface, why the signal could not have been correct, the fix, and the red/green evidence — flag any regression you introduced yourself in the same breath (one was found this way: a tightened bind killed a reverse-proxied vhost while every gate stayed green, because the spec covering that hostname did not exist).

Testing Policy — the seam inventory this audit walks
Observability — the scrape/probe surfaces it audits
Quadlet Lifecycle — the ownership labels the probes read

6.9 KiB Raw Blame History