30 lines
3.0 KiB
Markdown
30 lines
3.0 KiB
Markdown
---
|
|
name: fleet-service-runtime-census
|
|
description: "Answer \"what is service running right now?\" for a dank-fleet service: unit state, container image/digest, endpoint contract, loaded env — machine over repo, in that order."
|
|
---
|
|
|
|
Answer "what is service running right now?" for a dank-fleet service without trusting the repo. Repo state is intent; the machine is truth, and they differ by however long since the last deploy.
|
|
|
|
## Order of operations (cheap → expensive)
|
|
|
|
1. **Unit state on the host** (rootless quadlet ⇒ `--user`; some units are root):
|
|
`ssh -o BatchMode=yes host 'systemctl --user show S -p ActiveState -p SubState -p ExecMainStartTimestamp -p NRestarts'`
|
|
Boot-loop tell: `NRestarts` climbing with a short `ExecMainStartTimestamp`.
|
|
2. **The container's own view**: `podman ps --filter name=S --format '{{.Image}} {{.Status}} {{.Names}}'`, then `podman inspect` for `Image` (tag **and** `RepoDigests`) and `Env`.
|
|
3. **The endpoint contract, from the service itself** — this is the only answer that matters to a consumer:
|
|
`curl -s host:port/v1/models` (served model id), `curl -s host:port/props` (`n_ctx`, `n_ctx_train`, caps), `curl -s -o /dev/null -w '%{http_code}' host:port/health`.
|
|
4. **Effective flags**: the live process is the truth. `systemctl --user cat S` shows `EnvironmentFile=`; read that file. A repo `scripts/S.env` can be ahead of, or behind, what the running unit loaded — and a unit restart is what re-reads it.
|
|
5. **Logs last**: `journalctl --user -u S -n 80 --no-pager` for the boot ledger / refusal messages, after you know whether it is even up.
|
|
|
|
## Facts that bite
|
|
|
|
- **A service is up ≠ configured how you think.** llama.cpp's `/v1/models` reports `max_model_len`, not the engine's per-slot ceiling; halogen reports `n_ctx` per slot. Read `/props` before quoting any window number.
|
|
- **`systemctl is-enabled` answers enablement, not activity.** A timerless service reports `linked`/`static`; `deploy.sh` that prints `ENABLE` for it is lying. Read `systemctl is-enabled --quiet` exit status, not the text.
|
|
- **`NRestarts: 0` with `ActiveState=activating` is a crash-loop only if the timestamps move**; check `ExecMainStartTimestamp` twice, 5 s apart.
|
|
- **Read-only is always allowed** from the dev box (`ssh -o BatchMode=yes`); state changes on remote boxes are not, without a per-run OK. Production services (`frank-archon`, `frank-dragoon`, `frank-zealot`) are never stopped/restarted to "see what happens".
|
|
- **Digest-pinned images**: "when was this image updated?" is answered by `podman inspect` `RepoDigests` + the registry API, not by the unit's start time (a restart re-uses the local digest).
|
|
|
|
## Report
|
|
|
|
One line per layer with the evidence: unit (`active (running)`, since, restarts), container (image + digest), endpoint (served id, `n_ctx`, health code), loaded env (file path + the knobs that matter), and explicitly which of those came from the machine versus the repo — that gap is the answer to most "why is it behaving like that" questions.
|