Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
159 lines
7.7 KiB
Markdown
159 lines
7.7 KiB
Markdown
---
|
|
type: reference
|
|
title: Homepage Status Probes
|
|
description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions."
|
|
tags: [homepage, status, probes, e2e, homepage-status]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Homepage Status Probes
|
|
|
|
The homepage is a static site —
|
|
`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
|
|
— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
|
|
`scripts/deploy-remote.sh` refuses other targets). A checker
|
|
(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
|
|
each card's target and writes `/srv/homepage/status.json` (mode **0644** so
|
|
Caddy can serve it — losing the mode 403s the page).
|
|
|
|
## The checker's vantage and its three false-down shapes
|
|
|
|
The checker probes TCP/HTTP **from the host netns**. It only has three states
|
|
— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
|
|
never render honestly and must be dropped or re-pointed, not "fixed".
|
|
|
|
1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
|
|
only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
|
|
card says down while PMS answers `200` over HTTPS. Fix:
|
|
`"method":"tcp"` (connect is the honest liveness signal; the public URL
|
|
stays HTTP because Caddy re-keys).
|
|
2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
|
|
compose→quadlet move moved the dashboard to `9119` and the webui to
|
|
`8787`; nothing listened on 8090, so a healthy stack read `down`.
|
|
3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
|
|
`systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
|
|
and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
|
|
netns-scoped, so from the host these are unreachable *by construction* —
|
|
no probe value makes them honest. Drop those cards.
|
|
|
|
A host-vantage probe can only reach a listener **published to the host**
|
|
(rootlessport/pasta); unpublished ports are invisible from the host netns.
|
|
See [[Container Host Reachability]] for the netns rules.
|
|
|
|
## Triage procedure
|
|
|
|
1. Read the card's probe target in `shared/homepage/services.json`; note
|
|
method (`tcp` vs `http`).
|
|
2. Does that port have a **host-visible listener**?
|
|
`podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
|
|
`podman inspect <c> --format '{{json .NetworkSettings.Networks}}
|
|
{{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
|
|
published nothing → structurally unprobeable from the host).
|
|
3. Probe from the **correct vantage**: from a bridge-net container the host
|
|
is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
|
|
`dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
|
|
which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
|
|
bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
|
|
they expand **inside** `podman exec bash -c`.
|
|
4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
|
|
SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
|
|
killer) vs false down (running + unreachable target).
|
|
5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
|
|
only exposure is an internal netns; never invent a probe target.
|
|
6. Verify by re-running the checker path, not by eyeballing: after deploy,
|
|
re-read `/srv/homepage/status.json` and diff the `down` list
|
|
before/after.
|
|
|
|
Removing a card's `probe` key without a code change crashes the checker
|
|
(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
|
|
fix. The `description` field is consumed only by `app.js` `descHTML()`, so
|
|
adding a `details` object (role/spec/inputs/use) is schema-safe:
|
|
`tests/homepage_config.bats` checks only name/category/section/probe.
|
|
|
|
## Local render check (before every deploy)
|
|
|
|
The page fetches `services.json` + `status.json` at load; `file://` is
|
|
CORS-blocked, so serve over HTTP:
|
|
|
|
```bash
|
|
cd shared/homepage
|
|
python3 -c "import json; s=json.load(open('services.json')); \
|
|
json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
|
|
python3 -m http.server 8899 --bind 127.0.0.1
|
|
```
|
|
|
|
Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
|
|
(panels render after the async fetch; checking immediately returns 0 panels),
|
|
extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
|
|
`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
|
|
plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
|
|
visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
|
|
time) — harmless.
|
|
|
|
The scratch `status.json` is untracked and NOT git-ignored — move it to
|
|
`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
|
|
|
|
CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
|
|
`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
|
|
`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
|
|
reset so `?bare=1` strips it to black.
|
|
|
|
## Deploy
|
|
|
|
```
|
|
./scripts/deploy.sh dankclaw --service homepage --skip-e2e
|
|
```
|
|
|
|
Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
|
|
datetime), restarts the `homepage-status` timer. Verify live:
|
|
`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
|
|
|
|
**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
|
|
asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
|
|
Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
|
|
no remote wake) times out that test — a pre-existing environmental failure,
|
|
not a regression. All non-Mac-mini e2e tests pass.
|
|
|
|
## e2e raw-IP *arr failures — probe from the browser's box, not a peer
|
|
|
|
When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
|
|
Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
|
|
so a healthy probe from dankclaw proves nothing about the **browser's** path:
|
|
|
|
```bash
|
|
# 1. From the box that RUNS the browser:
|
|
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
|
|
# 2. From a LAN peer:
|
|
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
|
|
--max-time 6 http://192.168.0.112:$p/; done'
|
|
# 3. Liveness (ICMP, not TCP):
|
|
ping -c 2 -W 3 192.168.0.112
|
|
```
|
|
|
|
| peer probe | dev-box probe | ping | meaning |
|
|
|---|---|---|---|
|
|
| 200 | 200 | ok | transient — re-run the spec |
|
|
| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
|
|
| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
|
|
|
|
Third row: confirm the Mac is innocent through the dankclaw jump
|
|
(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
|
|
`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
|
|
--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
|
|
between the dev box and the Mac.
|
|
|
|
Don't hold a deploy hostage to a known-environmental red gate: verify the
|
|
deployed service **directly** (its own ledger/health/real completion), state
|
|
the gate was skipped or partially red and why (`--skip-e2e` prints the
|
|
warning, so the skip is recorded), and report the residual gap plainly
|
|
("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
|
|
probe returns 200; deploy verified by other means"). Do not weaken or skip
|
|
the spec — it guards a real SSO-plus-RBAC contract.
|
|
|
|
## Related
|
|
|
|
- [[Container Host Reachability]] — the netns rules behind the vantage
|
|
- [[Observability Provisioning]] — the monitoring stack reading the same fleet
|
|
- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
|
|
- [[Testing Policy]] — the seam/e2e contract this page's checks live in
|