Files

T

Frank 33eae71d33 docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

2026-10-05 21:19:15 -07:00

7.8 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Homepage Status Probes

The homepage is a static site — shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html} — served from /srv/homepage on dankclaw only (deploy_homepage in scripts/deploy-remote.sh refuses other targets). A checker (scripts/homepage-status.py, run by the homepage-status.timer unit) probes each card's target and writes /srv/homepage/status.json (mode 0644 so Caddy can serve it — losing the mode 403s the page).

The checker's vantage and its three false-down shapes

The checker probes TCP/HTTP from the host netns. It only has three states — wip | up | down; there is no "unknown", so an unprobeable service can never render honestly and must be dropped or re-pointed, not "fixed".

TLS-only app probed with method:"http". Plex Media Server serves TLS only on 32400; plain-HTTP GET → TLS alert → curl exit 52, code 000 → card says down while PMS answers 200 over HTTPS. Fix: "method":"tcp" (connect is the honest liveness signal; the public URL stays HTTP because Caddy re-keys).
Retired port. The Hermes card probed 127.0.0.1:8090 after the compose→quadlet move moved the dashboard to 9119 and the webui to 8787; nothing listened on 8090, so a healthy stack read down.
Unpublished port / wrong netns. gitea-postgres (db:5432, net systemd-gitea), searxng-valkey (valkey:6379, net systemd-searxng) and alloy (OTLP 12345) publish no ports. Container DNS aliases are netns-scoped, so from the host these are unreachable by construction — no probe value makes them honest. Drop those cards.

A host-vantage probe can only reach a listener published to the host (rootlessport/pasta); unpublished ports are invisible from the host netns. See Container Host Reachability for the netns rules.

Triage procedure

Read the card's probe target in shared/homepage/services.json; note method (tcp vs http).
Does that port have a host-visible listener? podman ps -a --format '{{.Names}}|{{.Status}}', then per container podman inspect <c> --format '{{json .NetworkSettings.Networks}} {{.HostConfig.NetworkMode}}' and podman port <c> (empty output = published nothing → structurally unprobeable from the host).
Probe from the correct vantage: from a bridge-net container the host is its gateway IP (/proc/net/route, e.g. 10.89.1.1 on dankclaw_default) — not host.containers.internal (169.254.1.2), which belongs to the default podman net (10.88.0.0/16). Probe with bash /dev/tcp (no python3/ip in these images); escape loop vars so they expand inside podman exec bash -c.
Classify: real outage (container exited — read State.ExitCode: 137 = SIGKILL, 143 = SIGTERM/manual stop; OOMKilled=false rules out the OOM killer) vs false down (running + unreachable target).
Fix: re-point or re-method for HTTP/TLS mismatches; drop cards whose only exposure is an internal netns; never invent a probe target.
Verify by re-running the checker path, not by eyeballing: after deploy, re-read /srv/homepage/status.json and diff the down list before/after.

Removing a card's probe key without a code change crashes the checker (svc["probe"] KeyError) — dropping the whole entry is the only zero-code fix. The description field is consumed only by app.js descHTML(), so adding a details object (role/spec/inputs/use) is schema-safe: tests/homepage_config.bats checks only name/category/section/probe.

Local render check (before every deploy)

The page fetches services.json + status.json at load; file:// is CORS-blocked, so serve over HTTP:

cd shared/homepage
python3 -c "import json; s=json.load(open('services.json')); \
  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
python3 -m http.server 8899 --bind 127.0.0.1

Then in the browser: load http://127.0.0.1:8899/index.html, wait ~1.5 s (panels render after the async fetch; checking immediately returns 0 panels), extract via tab.evaluate — LLM cards render .desc-role/.desc-spec/ .desc-inputs/.desc-use from the card's details object, other cards a plain .desc; getComputedStyle(el).color confirms tones. Screenshot for visual proof. Expected local noise: deployed.js 404s (baked in at deploy time) — harmless.

The scratch status.json is untracked and NOT git-ignored — move it to shared/homepage/.AGENTTRASH/ after the check (repo policy: never rm).

CSS palette lives in style.css :root (--fg bright green, --dim dim, --head pale, --link cyan, --wip yellow, --down red). Add new tones as --* vars and include any new .desc-* class in the html[data-bare="1"] reset so ?bare=1 strips it to black.

Deploy

./scripts/deploy.sh dankclaw --service homepage --skip-e2e

Copies the assets to /srv/homepage, regenerates deployed.js (git commit datetime), restarts the homepage-status timer. Verify live: ssh dank@192.168.0.143 'grep … /srv/homepage/…'.

--skip-e2e is justified for homepage-only changes when the Mac mini is asleep: the e2e admin crawl (tests/e2e/specs/11-admin.spec.ts:24) walks Prowlarr at 192.168.0.112:8989 on FrankBook-Pro; an asleep mini (ARP STALE, no remote wake) times out that test — a pre-existing environmental failure, not a regression. All non-Mac-mini e2e tests pass.

e2e raw-IP *arr failures — probe from the browser's box, not a peer

When 11-admin fails with ERR_CONNECTION_TIMED_OUT, the instinct is "the Mac is asleep". Sometimes true; not always. Playwright runs on the dev box, so a healthy probe from dankclaw proves nothing about the browser's path:

# 1. From the box that RUNS the browser:
curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
# 2. From a LAN peer:
ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
  --max-time 6 http://192.168.0.112:$p/; done'
# 3. Liveness (ICMP, not TCP):
ping -c 2 -W 3 192.168.0.112

peer probe	dev-box probe	ping	meaning
200	200	ok	transient — re-run the spec
timeout	timeout	fails	Mac asleep; wake, ports return in seconds
200	rc=28	ok	path problem, not sleep — ICMP allowed, TCP blocked. Waking cannot help

Third row: confirm the Mac is innocent through the dankclaw jump (pgrep -fl socat; netstat -an | grep -c 8989; loopback curl http://127.0.0.1:8989/ → 200; /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate → disabled). Forwarders up + loopback 200 ⇒ the block is between the dev box and the Mac.

Don't hold a deploy hostage to a known-environmental red gate: verify the deployed service directly (its own ledger/health/real completion), state the gate was skipped or partially red and why (--skip-e2e prints the warning, so the skip is recorded), and report the residual gap plainly ("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer probe returns 200; deploy verified by other means"). Do not weaken or skip the spec — it guards a real SSO-plus-RBAC contract.

Container Host Reachability — the netns rules behind the vantage
Observability Provisioning — the monitoring stack reading the same fleet
Deploy State Triage — service-scoped deploys and the e2e gate
Testing Policy — the seam/e2e contract this page's checks live in

7.8 KiB Raw Blame History