serverAppConfigs/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Homepage Status Probes	How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions.	homepage status probes e2e homepage-status	2026-10-05T00:00:00-07:00

Homepage Status Probes

The homepage is a static site — shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html} — served from /srv/homepage on dankclaw only (deploy_homepage in scripts/deploy-remote.sh refuses other targets). A checker (scripts/homepage-status.py, run by the homepage-status.timer unit) probes each card's target and writes /srv/homepage/status.json (mode 0644 so Caddy can serve it — losing the mode 403s the page).

The checker's vantage and its three false-down shapes

The checker probes TCP/HTTP from the host netns. It only has three states — wip | up | down; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".

1. TLS-only app probed with method:"http". Plex Media Server serves TLS only on 32400; plain-HTTP GET → TLS alert → curl exit 52, code 000 → card says down while PMS answers 200 over HTTPS. Fix: "method":"tcp" (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
2. Retired port. The Hermes card probed 127.0.0.1:8090 after the compose→quadlet move moved the dashboard to 9119 and the webui to 8787; nothing listened on 8090, so a healthy stack read down.
3. Unpublished port / wrong netns. gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng) and alloy (OTLP 12345) publish no ports. Container DNS aliases are netns-scoped, so from the host these are unreachable by construction — no probe value makes them honest. Drop those cards.

A host-vantage probe can only reach a listener published to the host (rootlessport/pasta); unpublished ports are invisible from the host netns. See Container Host Reachability for the netns rules.

Triage procedure

1. Read the card's probe target in shared/homepage/services.json; note method (tcp vs http).
2. Does that port have a host-visible listener? podman ps -a --format '{{.Names}}|{{.Status}}', then per container podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}' and podman port <c> (empty output = published nothing → structurally unprobeable from the host).
3. Probe from the correct vantage: from a bridge-net container the host is its gateway IP (/proc/net/route, e.g. 10.89.1.1 on dankclaw_default) — not host.containers.internal (169.254.1.2) which belongs to the default podman net (10.88.0.0/16). Probe with bash /dev/tcp (no python3/ip in these images); escape loop vars so they expand inside podman exec bash -c.
4. Classify: real outage (container exited — read State.ExitCode: 137 = SIGKILL, 143 = SIGTERM/manual stop; OOMKilled=false rules out the OOM killer) vs false down (running + unreachable target).
5. Fix: re-point or re-method for HTTP/TLS mismatches; drop cards whose only exposure is an internal netns; never invent a probe target.
6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read /srv/homepage/status.json and diff the down list before/after.

Removing a card's probe key without a code change crashes the checker (svc["probe"] KeyError) — dropping the whole entry is the only zero-code fix. The description field is consumed only by app.js descHTML(), so adding a details object (role/spec/inputs/use) is schema-safe: tests/homepage_config.bats checks only name/category/section/probe.

Local render check (before every deploy)

The page fetches services.json + status.json at load; file:// is CORS-blocked, so serve over HTTP:

cd shared/homepage
python3 -c "import json; s=json.load(open('services.json')); \
  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
python3 -m http.server 8899 --bind 127.0.0.1

Then in the browser: load http://127.0.0.1:8899/index.html, wait ~1.5 s (panels render after the async fetch; checking immediately returns 0 panels) extract via tab.evaluate — LLM cards render .desc-role/.desc-spec/ .desc-inputs/.desc-use from the card's details object, other cards a plain .desc; getComputedStyle(el).color confirms tones. Screenshot for visual proof. Expected local noise: deployed.js 404s (baked in at deploy time) — harmless.

The scratch status.json is untracked and NOT git-ignored — move it to shared/homepage/.AGENTTRASH/ after the check (repo policy: never rm).

CSS palette lives in style.css :root (--fg bright green, --dim dim, --head pale, --link cyan, --wip yellow, --down red). Add new tones as --* vars and include any new .desc-* class in the html[data-bare="1"] reset so ?bare=1 strips it to black.

Deploy

./scripts/deploy.sh dankclaw --service homepage --skip-e2e

Copies the assets to /srv/homepage, regenerates deployed.js (git commit datetime), restarts the homepage-status timer. Verify live: ssh dank@192.168.0.143 'grep … /srv/homepage/…'.

--skip-e2e is justified for homepage-only changes when the Mac mini is asleep: the e2e admin crawl (tests/e2e/specs/11-admin.spec.ts:24) walks Prowlarr at 192.168.0.112:8989 on FrankBook-Pro; an asleep mini (ARP STALE, no remote wake) times out that test — a pre-existing environmental failure, not a regression. All non-Mac-mini e2e tests pass.

e2e raw-IP *arr failures — probe from the browser's box, not a peer

When 11-admin fails with ERR_CONNECTION_TIMED_OUT, the instinct is "the Mac is asleep". Sometimes true; not always. Playwright runs on the dev box, so a healthy probe from dankclaw proves nothing about the browser's path:

# 1. From the box that RUNS the browser:
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
# 2. From a LAN peer:
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
  --max-time 6 http://192.168.0.112:$p/; done'
# 3. Liveness (ICMP, not TCP):
ping -c 2 -W 3 192.168.0.112

peer probe	dev-box probe	ping	meaning
200	200	ok	transient — re-run the spec
timeout	timeout	fails	Mac asleep; wake, ports return in seconds
200	rc=28	ok	path problem, not sleep — ICMP allowed, TCP blocked. Waking cannot help

Third row: confirm the Mac is innocent through the dankclaw jump (pgrep -fl socat; netstat -an | grep -c 8989; loopback curl http://127.0.0.1:8989/ → 200; /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate → disabled). Forwarders up + loopback 200 ⇒ the block is between the dev box and the Mac.

Don't hold a deploy hostage to a known-environmental red gate: verify the deployed service directly (its own ledger/health/real completion), state the gate was skipped or partially red and why (--skip-e2e prints the warning, so the skip is recorded), and report the residual gap plainly ("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer probe returns 200; deploy verified by other means"). Do not weaken or skip the spec — it guards a real SSO-plus-RBAC contract.

Related

• Container Host Reachability — the netns rules behind the vantage
• Observability Provisioning — the monitoring stack reading the same fleet
• Deploy State Triage — service-scoped deploys and the e2e gate
• Testing Policy — the seam/e2e contract this page's checks live in