Frank
9bfa0be85d
- 31 new managed skills (anubis/arr/authentik/hermes/freetoken/podman/llm-swap runbooks etc.) now tracked. - Removed superseded skills: omp_harness_config, peon_ping_integration (replaced by user-authored peon-ping-* managed skills), rocmfp4-strix-benchmark-orchestration. - New memory-root skills: niri_compositor_ops, omp_git_workflow, omp_model_config, quadlet-media-migration, homelab-service-triage, hermes-dankclaw-ops, frank-default-large-ops, caddy-auth-dns-playbook.
33 lines
4.1 KiB
Markdown
33 lines
4.1 KiB
Markdown
---
|
|
name: homepage-status-probe-namespace-triage
|
|
description: "Triage a homepage status card (or any host-side TCP probe) reporting \"down\" for a service that is actually up: decide by network namespace whether the probe target is reachable at all before calling it an outage."
|
|
---
|
|
|
|
# Goal
|
|
|
|
Decide, in one pass, whether a `down` card on the homepage status page (`scripts/homepage-status.py`, run by `homepage-status.timer` on Dankclaw, probes TCP/HTTP from the **host** netns) is a real outage or a structurally unreachable probe target. The checker only has three states — `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".
|
|
|
|
# Why it goes wrong
|
|
|
|
A host-vantage TCP probe can only reach a listener that is **published to the host** (rootless podman publishes via rootlessport/pasta; unpublished ports are invisible from the host netns). Three recurring false-down shapes on this fleet:
|
|
|
|
1. **TLS-only app probed with `method:"http"`.** Plex Media Server 1.43 serves TLS only on `32400`; a plain-HTTP GET gets a TLS alert → curl exit 52 "empty reply from server", code `000` → card says down while PMS answers `200` over HTTPS. Fix = `"method":"tcp"` (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
|
|
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the compose→quadlet move moved the dashboard to `9119` and the webui to `8787`; nothing listened on 8090, so a fully healthy stack read `down`.
|
|
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`), and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are netns-scoped, so from the host these are unreachable *by construction* — no probe value makes them honest.
|
|
|
|
# Procedure
|
|
|
|
1. Read the card's probe target in `shared/homepage/services.json`; note method (`tcp` vs `http`).
|
|
2. Ask: does that port have a **host-visible listener**? `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container `podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output = published nothing → structurally unprobeable from the host).
|
|
3. Probe from the **correct vantage**. From a bridge-net container, the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on `dankclaw_default`), *not* `host.containers.internal` (`169.254.1.2`) — that pasta alias belongs to the default `podman` net (`10.88.0.0/16`, gw `10.88.0.1`) and is not routed from other nets. Probe with bash `/dev/tcp` (no `python3`/`ip` in these images), and escape the loop var so it expands **inside** `podman exec bash -c`.
|
|
4. Classify: real outage (container exited — read `State.ExitCode`: 137 = SIGKILL, 143 = SIGTERM/manual stop, `OOMKilled=false` rules out the OOM killer) vs false down (running + unreachable target).
|
|
5. Fix accordingly: re-point or re-method the probe for a real HTTP/TLS mismatch; **drop** cards whose only exposure is an internal netns (or accept permanent false-downs and say so); never invent a probe target for them.
|
|
6. Verify by re-running the checker path, not by eyeballing: after deploy, re-read `/srv/homepage/status.json` and diff the `down` list before/after.
|
|
|
|
# Watch out
|
|
|
|
- Deploying this file is service-scoped: `./scripts/run.sh` (233 tests) → commit → `./scripts/deploy.sh dankclaw --service homepage`; the deployer *pushes* `main` first, so any already-committed local work rides along — say so before pushing.
|
|
- The checker writes `/srv/homepage/status.json` world-readable (0644) so Caddy can serve it; keep that mode or Caddy 403s the page.
|
|
- Removing a card's `probe` key without code change crashes the checker (`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code fix.
|
|
- Bare compose services with `restart=unless-stopped` (node-exporter on both hosts) do **not** survive reboot: rootless podman does not autostart bare containers, and `deploy-remote.sh` refuses to `compose up` Dankserver's compose services by design → they need a quadlet unit, not a restart.
|