---
type: reference
title: Homepage Status Probes
description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions."
tags: [homepage, status, probes, e2e, homepage-status]
timestamp: 2026-10-05T00:00:00-07:00
---

# Homepage Status Probes

The homepage is a static site —
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
`scripts/deploy-remote.sh` refuses other targets). A checker
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
Caddy can serve it — losing the mode 403s the page).

## The checker's vantage and its three false-down shapes

The checker probes TCP/HTTP **from the host netns**. It only has three states
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
never render honestly and must be dropped or re-pointed, not "fixed".

1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
   card says down while PMS answers `200` over HTTPS. Fix:
   `"method":"tcp"` (connect is the honest liveness signal; the public URL
   stays HTTP because Caddy re-keys).
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
   compose→quadlet move moved the dashboard to `9119` and the webui to
   `8787`; nothing listened on 8090, so a healthy stack read `down`.
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
   netns-scoped, so from the host these are unreachable *by construction* —
   no probe value makes them honest. Drop those cards.

A host-vantage probe can only reach a listener **published to the host**
(rootlessport/pasta); unpublished ports are invisible from the host netns.
See [[Container Host Reachability]] for the netns rules.

## Triage procedure

1. Read the card's probe target in `shared/homepage/services.json`; note
   method (`tcp` vs `http`).
2. Does that port have a **host-visible listener**?
   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
   published nothing → structurally unprobeable from the host).
3. Probe from the **correct vantage**: from a bridge-net container the host
   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
   they expand **inside** `podman exec bash -c`.
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
   killer) vs false down (running + unreachable target).
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
   only exposure is an internal netns; never invent a probe target.
6. Verify by re-running the checker path, not by eyeballing: after deploy,
   re-read `/srv/homepage/status.json` and diff the `down` list
   before/after.

Removing a card's `probe` key without a code change crashes the checker
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
adding a `details` object (role/spec/inputs/use) is schema-safe:
`tests/homepage_config.bats` checks only name/category/section/probe.

## Local render check (before every deploy)

The page fetches `services.json` + `status.json` at load; `file://` is
CORS-blocked, so serve over HTTP:

```bash
cd shared/homepage
python3 -c "import json; s=json.load(open('services.json')); \
  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
python3 -m http.server 8899 --bind 127.0.0.1
```

Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
(panels render after the async fetch; checking immediately returns 0 panels)
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
time) — harmless.

The scratch `status.json` is untracked and NOT git-ignored — move it to
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).

CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
reset so `?bare=1` strips it to black.

## Deploy

```
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
```

Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
datetime), restarts the `homepage-status` timer. Verify live:
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.

**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
no remote wake) times out that test — a pre-existing environmental failure,
not a regression. All non-Mac-mini e2e tests pass.

## e2e raw-IP *arr failures — probe from the browser's box, not a peer

When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
so a healthy probe from dankclaw proves nothing about the **browser's** path:

```bash
# 1. From the box that RUNS the browser:
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
# 2. From a LAN peer:
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
  --max-time 6 http://192.168.0.112:$p/; done'
# 3. Liveness (ICMP, not TCP):
ping -c 2 -W 3 192.168.0.112
```

| peer probe | dev-box probe | ping | meaning |
|---|---|---|---|
| 200 | 200 | ok | transient — re-run the spec |
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |

Third row: confirm the Mac is innocent through the dankclaw jump
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
between the dev box and the Mac.

Don't hold a deploy hostage to a known-environmental red gate: verify the
deployed service **directly** (its own ledger/health/real completion), state
the gate was skipped or partially red and why (`--skip-e2e` prints the
warning, so the skip is recorded), and report the residual gap plainly
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
probe returns 200; deploy verified by other means"). Do not weaken or skip
the spec — it guards a real SSO-plus-RBAC contract.

## Related

- [[Container Host Reachability]] — the netns rules behind the vantage
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
- [[Testing Policy]] — the seam/e2e contract this page's checks live in