command-center/agent/managed-skills/homepage-status-probe-namespace-triage/SKILL.md

name, description

name	description
homepage-status-probe-namespace-triage	Triage a homepage status card (or any host-side TCP probe) reporting "down" for a service that is actually up: decide by network namespace whether the probe target is reachable at all before calling it an outage.

Goal

Decide, in one pass, whether a down card on the homepage status page (scripts/homepage-status.py, run by homepage-status.timer on Dankclaw, probes TCP/HTTP from the host netns) is a real outage or a structurally unreachable probe target. The checker only has three states — wip | up | down; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".

Why it goes wrong

A host-vantage TCP probe can only reach a listener that is published to the host (rootless podman publishes via rootlessport/pasta; unpublished ports are invisible from the host netns). Three recurring false-down shapes on this fleet:

1. TLS-only app probed with method:"http". Plex Media Server 1.43 serves TLS only on 32400; a plain-HTTP GET gets a TLS alert → curl exit 52 "empty reply from server", code 000 → card says down while PMS answers 200 over HTTPS. Fix = "method":"tcp" (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
2. Retired port. The Hermes card probed 127.0.0.1:8090 after the compose→quadlet move moved the dashboard to 9119 and the webui to 8787; nothing listened on 8090, so a fully healthy stack read down.
3. Unpublished port / wrong netns. gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng), and alloy (OTLP 12345) publish no ports. Container DNS aliases are netns-scoped, so from the host these are unreachable by construction — no probe value makes them honest.

Procedure

1. Read the card's probe target in shared/homepage/services.json; note method (tcp vs http).
2. Ask: does that port have a host-visible listener? podman ps -a --format '{{.Names}}|{{.Status}}', then per container podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}' and podman port <c> (empty output = published nothing → structurally unprobeable from the host).
3. Probe from the correct vantage. From a bridge-net container, the host is its gateway IP (/proc/net/route, e.g. 10.89.1.1 on dankclaw_default), not host.containers.internal (169.254.1.2) — that pasta alias belongs to the default podman net (10.88.0.0/16, gw 10.88.0.1) and is not routed from other nets. Probe with bash /dev/tcp (no python3/ip in these images), and escape the loop var so it expands inside podman exec bash -c.
4. Classify: real outage (container exited — read State.ExitCode: 137 = SIGKILL, 143 = SIGTERM/manual stop, OOMKilled=false rules out the OOM killer) vs false down (running + unreachable target).
5. Fix accordingly: re-point or re-method the probe for a real HTTP/TLS mismatch; drop cards whose only exposure is an internal netns (or accept permanent false-downs and say so); never invent a probe target for them.
6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read /srv/homepage/status.json and diff the down list before/after.

Watch out

• Deploying this file is service-scoped: ./scripts/run.sh (233 tests) → commit → ./scripts/deploy.sh dankclaw --service homepage; the deployer pushes main first, so any already-committed local work rides along — say so before pushing.
• The checker writes /srv/homepage/status.json world-readable (0644) so Caddy can serve it; keep that mode or Caddy 403s the page.
• Removing a card's probe key without code change crashes the checker (svc["probe"] KeyError) — dropping the whole entry is the only zero-code fix.
• Bare compose services with restart=unless-stopped (node-exporter on both hosts) do not survive reboot: rootless podman does not autostart bare containers, and deploy-remote.sh refuses to compose up Dankserver's compose services by design → they need a quadlet unit, not a restart.