Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
159 lines
7.7 KiB
Markdown
159 lines
7.7 KiB
Markdown
---
|
|
type: reference
|
|
title: Homepage Status Probes
|
|
description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions."
|
|
tags: [homepage, status, probes, e2e, homepage-status]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Homepage Status Probes
|
|
|
|
The homepage is a static site —
|
|
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
|
|
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
|
|
`scripts/deploy-remote.sh` refuses other targets). A checker
|
|
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
|
|
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
|
|
Caddy can serve it — losing the mode 403s the page).
|
|
|
|
## The checker's vantage and its three false-down shapes
|
|
|
|
The checker probes TCP/HTTP **from the host netns**. It only has three states
|
|
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
|
|
never render honestly and must be dropped or re-pointed, not "fixed".
|
|
|
|
1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
|
|
only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
|
|
card says down while PMS answers `200` over HTTPS. Fix:
|
|
`"method":"tcp"` (connect is the honest liveness signal; the public URL
|
|
stays HTTP because Caddy re-keys).
|
|
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
|
|
compose→quadlet move moved the dashboard to `9119` and the webui to
|
|
`8787`; nothing listened on 8090, so a healthy stack read `down`.
|
|
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
|
|
`systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
|
|
and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
|
|
netns-scoped, so from the host these are unreachable *by construction* —
|
|
no probe value makes them honest. Drop those cards.
|
|
|
|
A host-vantage probe can only reach a listener **published to the host**
|
|
(rootlessport/pasta); unpublished ports are invisible from the host netns.
|
|
See [[Container Host Reachability]] for the netns rules.
|
|
|
|
## Triage procedure
|
|
|
|
1. Read the card's probe target in `shared/homepage/services.json`; note
|
|
method (`tcp` vs `http`).
|
|
2. Does that port have a **host-visible listener**?
|
|
`podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
|
|
`podman inspect <c> --format '{{json .NetworkSettings.Networks}}
|
|
{{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
|
|
published nothing → structurally unprobeable from the host).
|
|
3. Probe from the **correct vantage**: from a bridge-net container the host
|
|
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
|
|
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
|
|
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
|
|
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
|
|
they expand **inside** `podman exec bash -c`.
|
|
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
|
|
SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
|
|
killer) vs false down (running + unreachable target).
|
|
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
|
|
only exposure is an internal netns; never invent a probe target.
|
|
6. Verify by re-running the checker path, not by eyeballing: after deploy,
|
|
re-read `/srv/homepage/status.json` and diff the `down` list
|
|
before/after.
|
|
|
|
Removing a card's `probe` key without a code change crashes the checker
|
|
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
|
|
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
|
|
adding a `details` object (role/spec/inputs/use) is schema-safe:
|
|
`tests/homepage_config.bats` checks only name/category/section/probe.
|
|
|
|
## Local render check (before every deploy)
|
|
|
|
The page fetches `services.json` + `status.json` at load; `file://` is
|
|
CORS-blocked, so serve over HTTP:
|
|
|
|
```bash
|
|
cd shared/homepage
|
|
python3 -c "import json; s=json.load(open('services.json')); \
|
|
json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
|
|
python3 -m http.server 8899 --bind 127.0.0.1
|
|
```
|
|
|
|
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
|
|
(panels render after the async fetch; checking immediately returns 0 panels)
|
|
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
|
|
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
|
|
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
|
|
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
|
|
time) — harmless.
|
|
|
|
The scratch `status.json` is untracked and NOT git-ignored — move it to
|
|
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
|
|
|
|
CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
|
|
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
|
|
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
|
|
reset so `?bare=1` strips it to black.
|
|
|
|
## Deploy
|
|
|
|
```
|
|
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
|
|
```
|
|
|
|
Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
|
|
datetime), restarts the `homepage-status` timer. Verify live:
|
|
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
|
|
|
|
**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
|
|
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
|
|
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
|
|
no remote wake) times out that test — a pre-existing environmental failure,
|
|
not a regression. All non-Mac-mini e2e tests pass.
|
|
|
|
## e2e raw-IP *arr failures — probe from the browser's box, not a peer
|
|
|
|
When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
|
|
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
|
|
so a healthy probe from dankclaw proves nothing about the **browser's** path:
|
|
|
|
```bash
|
|
# 1. From the box that RUNS the browser:
|
|
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
|
|
# 2. From a LAN peer:
|
|
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
|
|
--max-time 6 http://192.168.0.112:$p/; done'
|
|
# 3. Liveness (ICMP, not TCP):
|
|
ping -c 2 -W 3 192.168.0.112
|
|
```
|
|
|
|
| peer probe | dev-box probe | ping | meaning |
|
|
|---|---|---|---|
|
|
| 200 | 200 | ok | transient — re-run the spec |
|
|
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
|
|
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
|
|
|
|
Third row: confirm the Mac is innocent through the dankclaw jump
|
|
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
|
|
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
|
|
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
|
|
between the dev box and the Mac.
|
|
|
|
Don't hold a deploy hostage to a known-environmental red gate: verify the
|
|
deployed service **directly** (its own ledger/health/real completion), state
|
|
the gate was skipped or partially red and why (`--skip-e2e` prints the
|
|
warning, so the skip is recorded), and report the residual gap plainly
|
|
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
|
|
probe returns 200; deploy verified by other means"). Do not weaken or skip
|
|
the spec — it guards a real SSO-plus-RBAC contract.
|
|
|
|
## Related
|
|
|
|
- [[Container Host Reachability]] — the netns rules behind the vantage
|
|
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
|
|
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
|
|
- [[Testing Policy]] — the seam/e2e contract this page's checks live in
|