live: heartbeat sync 2026-10-21T02:32:51-07:00 (Refs #11)

agent/managed-skills/fleet-service-runtime-census/SKILL.md

@@ -1,62 +1,29 @@
 ---
 name: fleet-service-runtime-census
-description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins."
+description: "Answer \"what is service running right now?\" for a dank-fleet service: unit state, container image/digest, endpoint contract, loaded env — machine over repo, in that order."
 ---
 
-# Fleet service runtime census
+Answer "what is service running right now?" for a dank-fleet service without trusting the repo. Repo state is intent; the machine is truth, and they differ by however long since the last deploy.
 
-Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table.
+## Order of operations (cheap → expensive)
 
-## 1. Probe the published port (works from any LAN box, no ssh)
+1. **Unit state on the host** (rootless quadlet ⇒ `--user`; some units are root):
+   `ssh -o BatchMode=yes host 'systemctl --user show S -p ActiveState -p SubState -p ExecMainStartTimestamp -p NRestarts'`
+   Boot-loop tell: `NRestarts` climbing with a short `ExecMainStartTimestamp`.
+2. **The container's own view**: `podman ps --filter name=S --format '{{.Image}} {{.Status}} {{.Names}}'`, then `podman inspect` for `Image` (tag **and** `RepoDigests`) and `Env`.
+3. **The endpoint contract, from the service itself** — this is the only answer that matters to a consumer:
+   `curl -s host:port/v1/models` (served model id), `curl -s host:port/props` (`n_ctx`, `n_ctx_train`, caps), `curl -s -o /dev/null -w '%{http_code}' host:port/health`.
+4. **Effective flags**: the live process is the truth. `systemctl --user cat S` shows `EnvironmentFile=`; read that file. A repo `scripts/S.env` can be ahead of, or behind, what the running unit loaded — and a unit restart is what re-reads it.
+5. **Logs last**: `journalctl --user -u S -n 80 --no-pager` for the boot ledger / refusal messages, after you know whether it is even up.
 
-```bash
-curl -s -m 10 http://<host-ip>:<port>/health; echo
-curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices   # per-endpoint extras: /v1/models, /metrics
-```
+## Facts that bite
 
-The extra route matters more than `/health` for identity claims:
-- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`.
-- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`.
-- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.
+- **A service is up ≠ configured how you think.** llama.cpp's `/v1/models` reports `max_model_len`, not the engine's per-slot ceiling; halogen reports `n_ctx` per slot. Read `/props` before quoting any window number.
+- **`systemctl is-enabled` answers enablement, not activity.** A timerless service reports `linked`/`static`; `deploy.sh` that prints `ENABLE` for it is lying. Read `systemctl is-enabled --quiet` exit status, not the text.
+- **`NRestarts: 0` with `ActiveState=activating` is a crash-loop only if the timestamps move**; check `ExecMainStartTimestamp` twice, 5 s apart.
+- **Read-only is always allowed** from the dev box (`ssh -o BatchMode=yes`); state changes on remote boxes are not, without a per-run OK. Production services (`frank-archon`, `frank-dragoon`, `frank-zealot`) are never stopped/restarted to "see what happens".
+- **Digest-pinned images**: "when was this image updated?" is answered by `podman inspect` `RepoDigests` + the registry API, not by the unit's start time (a restart re-uses the local digest).
 
-## 2. Units + containers on the host (ssh as the deploy user, no sudo)
+## Report
 
-```bash
-ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
-  | grep -E "^(●|  Active|  Loaded)"; \
-  podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
-  podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
-```
-
-- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
-- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
-
-## 3. Drop-ins — the override that no health gate sees
-
-```bash
-ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
-            systemctl --user show <engine>.service -p DropInPaths'
-```
-
-`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
-
-## 4. Live env vs repo env — the only truth test
-
-```bash
-# repo side (local checkout)
-grep -vE '^\s*#|^\s*$' scripts/<svc>.env
-# host side
-ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
-```
-
-Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
-
-## 5. Report
-
-Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
-
-## Traps that produce false findings
-
-- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
-- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
-- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
+One line per layer with the evidence: unit (`active (running)`, since, restarts), container (image + digest), endpoint (served id, `n_ctx`, health code), loaded env (file path + the knobs that matter), and explicitly which of those came from the machine versus the repo — that gap is the answer to most "why is it behaving like that" questions.