Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

6.8 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

Stale Signal Audit

Run after any change and before trusting a green gate. The productive question is which signal here cannot possibly be correct? Every pass on this repo (4 of 4) found real defects — including one in the rule file every future agent reads first.

The surfaces, and what each vantage point can reach

Surface	Producer's vantage point	Can reach
Homepage status cards	the `homepage-status` timer on Dankclaw	Dankclaw's loopback, Dankclaw container names, published LAN addresses
Prometheus scrape jobs	the `prometheus` container on Dankclaw	container names on `dankclaw_default`, LAN addresses, that host's loopback only
bats / e2e suites	the dev box	the repo, plus the live fleet over HTTPS/LAN

Anything else — another host's loopback, a container-internal port never published — is unreachable no matter how healthy the service is. The failure mode is always the same shape: a signal that cannot succeed by construction, which teaches the operator to ignore the surface it lives on.

1. Permanently-red probes (reachability)

For every monitor/probe/alert target ask: can this checker physically reach that endpoint? Seen classes: a card probing another host's container-internal port; a "probe" job scraping / (HTML → unsupported Content-Type); a /metrics that is disabled (Gitea 404).

A scrape cannot express "is this HTTP service answering?" — the right probe is the blackbox exporter in probe mode (/probe?target=…&module=http_2xx), which also gives guarded vhosts a real uptime signal, since Prometheus cannot solve an Anubis challenge.
Fix in the direction of honesty, never of silence: unreachable target → mark the homepage card wip: true with the reason in description (the checker skips wip; wip is the existing convention for "real service, not probeable"), or repoint the job at something reachable. NEVER open a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable.
Alert rules must not depend on always-red signals — scope them to services that can actually be healthy; the frank-* scope LlmEndpointDown carries is recorded on Prometheus.
A target's failure STRING can prove a fix: a /metrics scrape flipping from connection refused to unsupported Content-Type "text/html" means the service came up and only the scrape is wrong. Read the error, not just up/down.

2. Vacuous assertions (highest severity — the gate is already dead)

A seam that greps a file passes by absence once the file is retired: grep exits 2 on a missing path and ! inverts it.

grep -rn 'grep.*"\$REPO_ROOT/[^"]*"' tests/*.bats | head -50
[ -f "$path" ] || echo "MISSING: a seam may be grepping this"

Fix shape — one place asserts the invariant, each local site keeps a live check:

@test "no <host> compose file competes with the quadlet units" {
    local cf="$REPO_ROOT/<host>/compose.yaml"
    [ ! -f "$cf" ] || { echo "$cf is back: every container is unit-managed"; return 1; }
}
# at the per-service site: still fails if the entry returns
[ ! -f "$cf" ] || ! grep -qE '^\s+<svc>:' "$cf"

Prove teeth: recreate the offending file (printf 'services:\n <svc>:\n image: x\n' > <path>), run the seams, confirm RED, then move the file to trash. A re-armed check you have never seen fail is still unverified.

3. Gates that never run

Playwright specs in tests/e2e/ are matched by number prefix in playwright.config.ts; an unregistered spec silently never runs and the suite count stays the same. Count tests before/after; tests/e2e_scaffold.bats fails on an unregistered spec by design.

4. Validate the checker before acting on it

A checker's output is a hypothesis about another tool's semantics. Modelling Obsidian with naive text matching produced 45 "broken links" of which 3 were real — the rest were case-sensitivity (a qBittorrent link to qbittorrent.md), path-qualified resolution (a path-qualified folder/Note link resolves by path SUFFIX, not stem), and %20 URL-encoding in markdown links. Fix the checker first, re-derive the report, only then edit documents. The result lives in scripts/check-vault-links.py with those three rules in its docstring, gated by the no unresolved links in the docs vault seam. Same class: a find … | head -1 selector that reads the wrong file; a truncated listing that makes you "fix" a file that exists.

5. Prose invalidated by your own change

After retiring a file or adding a feature, grep for EVERY reference, not just code:

grep -rn "<retired-file>" --include="*" . | grep -v '^./.git/'
grep -rn "no <feature>\|none —\|unconfigured" --include="*.md" --include="*.container" --include="*.yml" .

This catches the ones that bite: AGENTS.md said ComfyUI runs CPU-only long after it was restored with GPU access; a unit said "there are no alerting rules" hours after five were added; a docs page documented three scrape jobs as working when they could never report up. Clean cutover includes prose — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale).

Order and verification

Enumerate the surfaces (status board, metrics targets, alert rules, test gates, docs/rule files, dashboard panels).
For each, ask what the signal CAN express, not what it says.
Validate any checker before editing the things it flags.
Fix at the root; never weaken an assertion to make it pass.
Gate the class so it cannot return (a seam), then prove the seam fails on the defect.

Verify: ./scripts/check-vault-links.py (exit 1 on breaks); query each Prometheus panel expression against the API (0 series on a panel that should show data is a defect; 0 ALERTS when nothing fires is not); read the deployed /srv/homepage/status.json and triage every state != up entry against reality; query /api/v1/targets?state=active and print health != "up" with lastError; ./tests/run.sh → ALL TESTS PASSED. Report per finding: the surface, why the signal could not have been correct, the fix, and the red/green evidence — flag any regression you introduced yourself in the same breath (one was found this way: a tightened bind killed a reverse-proxied vhost while every gate stayed green, because the spec covering that hostname did not exist).

Testing Policy — the seam inventory this audit walks
Observability — the scrape/probe surfaces it audits
Quadlet Lifecycle — the ownership labels the probes read

6.8 KiB Raw Permalink Blame History