Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

7.7 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Homepage Status Probes

The homepage is a static site — shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html} — served from /srv/homepage on dankclaw only (deploy_homepage in scripts/deploy-remote.sh refuses other targets). A checker (scripts/homepage-status.py, run by the homepage-status.timer unit) probes each card's target and writes /srv/homepage/status.json (mode 0644 so Caddy can serve it — losing the mode 403s the page).

The checker's vantage and its three false-down shapes

The checker probes TCP/HTTP from the host netns. It only has three states — wip | up | down; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".

TLS-only app probed with method:"http". Plex Media Server serves TLS only on 32400; plain-HTTP GET → TLS alert → curl exit 52, code 000 → card says down while PMS answers 200 over HTTPS. Fix: "method":"tcp" (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
Retired port. The Hermes card probed 127.0.0.1:8090 after the compose→quadlet move moved the dashboard to 9119 and the webui to 8787; nothing listened on 8090, so a healthy stack read down.
Unpublished port / wrong netns. gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng) and alloy (OTLP 12345) publish no ports. Container DNS aliases are netns-scoped, so from the host these are unreachable by construction — no probe value makes them honest. Drop those cards.

A host-vantage probe can only reach a listener published to the host (rootlessport/pasta); unpublished ports are invisible from the host netns. See Container Host Reachability for the netns rules.

Triage procedure

Read the card's probe target in shared/homepage/services.json; note method (tcp vs http).
Does that port have a host-visible listener? podman ps -a --format '{{.Names}}|{{.Status}}', then per container podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}' and podman port <c> (empty output = published nothing → structurally unprobeable from the host).
Probe from the correct vantage: from a bridge-net container the host is its gateway IP (/proc/net/route, e.g. 10.89.1.1 on dankclaw_default) — not host.containers.internal (169.254.1.2), which belongs to the default podman net (10.88.0.0/16). Probe with bash /dev/tcp (no python3/ip in these images); escape loop vars so they expand inside podman exec bash -c.
Classify: real outage (container exited — read State.ExitCode: 137 = SIGKILL, 143 = SIGTERM/manual stop; OOMKilled=false rules out the OOM killer) vs false down (running + unreachable target).
Fix: re-point or re-method for HTTP/TLS mismatches; drop cards whose only exposure is an internal netns; never invent a probe target.
Verify by re-running the checker path, not by eyeballing: after deploy, re-read /srv/homepage/status.json and diff the down list before/after.

Removing a card's probe key without a code change crashes the checker (svc["probe"] KeyError) — dropping the whole entry is the only zero-code fix. The description field is consumed only by app.js descHTML(), so adding a details object (role/spec/inputs/use) is schema-safe: tests/homepage_config.bats checks only name/category/section/probe.

Local render check (before every deploy)

The page fetches services.json + status.json at load; file:// is CORS-blocked, so serve over HTTP:

cd shared/homepage
python3 -c "import json; s=json.load(open('services.json')); \
  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
python3 -m http.server 8899 --bind 127.0.0.1

Then in the browser: load http://127.0.0.1:8899/index.html, wait ~1.5 s (panels render after the async fetch; checking immediately returns 0 panels), extract via tab.evaluate — LLM cards render .desc-role/.desc-spec/ .desc-inputs/.desc-use from the card's details object, other cards a plain .desc; getComputedStyle(el).color confirms tones. Screenshot for visual proof. Expected local noise: deployed.js 404s (baked in at deploy time) — harmless.

The scratch status.json is untracked and NOT git-ignored — move it to shared/homepage/.AGENTTRASH/ after the check (repo policy: never rm).

CSS palette lives in style.css :root (--fg bright green, --dim dim, --head pale, --link cyan, --wip yellow, --down red). Add new tones as --* vars and include any new .desc-* class in the html[data-bare="1"] reset so ?bare=1 strips it to black.

Deploy

./scripts/deploy.sh dankclaw --service homepage --skip-e2e

Copies the assets to /srv/homepage, regenerates deployed.js (git commit datetime), restarts the homepage-status timer. Verify live: ssh dank@192.168.0.143 'grep … /srv/homepage/…'.

--skip-e2e is justified for homepage-only changes when the Mac mini is asleep: the e2e admin crawl (tests/e2e/specs/11-admin.spec.ts:24) walks Prowlarr at 192.168.0.112:8989 on FrankBook-Pro; an asleep mini (ARP STALE, no remote wake) times out that test — a pre-existing environmental failure, not a regression. All non-Mac-mini e2e tests pass.

e2e raw-IP *arr failures — probe from the browser's box, not a peer

When 11-admin fails with ERR_CONNECTION_TIMED_OUT, the instinct is "the Mac is asleep". Sometimes true; not always. Playwright runs on the dev box, so a healthy probe from dankclaw proves nothing about the browser's path:

# 1. From the box that RUNS the browser:
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
# 2. From a LAN peer:
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
  --max-time 6 http://192.168.0.112:$p/; done'
# 3. Liveness (ICMP, not TCP):
ping -c 2 -W 3 192.168.0.112

peer probe	dev-box probe	ping	meaning
200	200	ok	transient — re-run the spec
timeout	timeout	fails	Mac asleep; wake, ports return in seconds
200	rc=28	ok	path problem, not sleep — ICMP allowed, TCP blocked. Waking cannot help

Third row: confirm the Mac is innocent through the dankclaw jump (pgrep -fl socat; netstat -an | grep -c 8989; loopback curl http://127.0.0.1:8989/ → 200; /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate → disabled). Forwarders up + loopback 200 ⇒ the block is between the dev box and the Mac.

Don't hold a deploy hostage to a known-environmental red gate: verify the deployed service directly (its own ledger/health/real completion), state the gate was skipped or partially red and why (--skip-e2e prints the warning, so the skip is recorded), and report the residual gap plainly ("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer probe returns 200; deploy verified by other means"). Do not weaken or skip the spec — it guards a real SSO-plus-RBAC contract.

Container Host Reachability — the netns rules behind the vantage
Observability Provisioning — the monitoring stack reading the same fleet
Deploy State Triage — service-scoped deploys and the e2e gate
Testing Policy — the seam/e2e contract this page's checks live in

7.7 KiB Raw Blame History