63 lines
3.4 KiB
Markdown
63 lines
3.4 KiB
Markdown
---
|
|
name: fleet-service-runtime-census
|
|
description: "Answer \"what is service running right now?\" for a dank-fleet service: live endpoint probe plus host units, image ids, live-vs-repo env, and drop-ins."
|
|
---
|
|
|
|
# Fleet service runtime census
|
|
|
|
Answer the question "what is `<service>` running currently?" with evidence, not from docs. Docs say what *should* run; the host says what *does*. Run all five probes, then report one table.
|
|
|
|
## 1. Probe the published port (works from any LAN box, no ssh)
|
|
|
|
```bash
|
|
curl -s -m 10 http://<host-ip>:<port>/health; echo
|
|
curl -s -m 10 http://<host-ip>:<port>/v1/audio/voices # per-endpoint extras: /v1/models, /metrics
|
|
```
|
|
|
|
The extra route matters more than `/health` for identity claims:
|
|
- llama.cpp and radiance **echo the requested model name** — a 200 proves nothing; read `/v1/models`.
|
|
- halogen 404s `/completion`, `/ping`, `/props`, `/tokenize`.
|
|
- The frank-tts façade answers `/health` **without dialing the engine**, so "façade healthy, engine down" is a real state; the voice list in that same response is the honest census.
|
|
|
|
## 2. Units + containers on the host (ssh as the deploy user, no sudo)
|
|
|
|
```bash
|
|
ssh <host> 'systemctl --user status <svc>.service <engine>.service --no-pager \
|
|
| grep -E "^(●| Active| Loaded)"; \
|
|
podman images --format "{{.Repository}}:{{.Tag}} {{.ID}} {{.Created}}"; \
|
|
podman ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"'
|
|
```
|
|
|
|
- `active (running) since …` + `Up N hours (healthy)` is the answer; `activating (auto-restart)` with a ~50 ms exit is a **preflight refusal**, not an engine fault.
|
|
- **NEVER prefix these podman calls with `sudo`.** Containers live in the deploy user's rootless storage; `sudo podman inspect frank-tts-engine` returns `no such object` while the plain form succeeds. Treating that as "container missing" is a false finding.
|
|
|
|
## 3. Drop-ins — the override that no health gate sees
|
|
|
|
```bash
|
|
ssh <host> 'systemctl --user show <svc>.service -p DropInPaths; \
|
|
systemctl --user show <engine>.service -p DropInPaths'
|
|
```
|
|
|
|
`DropInPaths=` must be empty. A leftover bench drop-in (`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`) makes the next deploy boot a different model at a different shape while `/health` still answers 200 and the e2e gate passes.
|
|
|
|
## 4. Live env vs repo env — the only truth test
|
|
|
|
```bash
|
|
# repo side (local checkout)
|
|
grep -vE '^\s*#|^\s*$' scripts/<svc>.env
|
|
# host side
|
|
ssh <host> 'sudo cat /home/dank/serverAppConfigs/scripts/<svc>.env' | grep -vE '^\s*#|^\s*$'
|
|
```
|
|
|
|
Diff them. Per repo law the env file beside the serve script **is** the authority; a doc page that disagrees with it is the bug, and host-side drift means an uncommitted edit. Also check `git log --oneline -3 -- scripts/<svc>.env` to confirm the host isn't ahead of git.
|
|
|
|
## 5. Report
|
|
|
|
Table of: unit state + since, image repo:tag@short-ID + age, backend/alias actually served (from step 1, not from the name), drop-ins, and whether live env == repo env. State explicitly when a fact came from docs rather than from a probe.
|
|
|
|
## Traps that produce false findings
|
|
|
|
- Digest-pinned images: the local image ID's *creation date* answers "is this stale" better than the tag; `latest` + `Pull=never` is a locally built image, not a registry pull.
|
|
- `ssh` user context changes what podman sees (root vs rootless) — see step 2.
|
|
- Two ports on one host are two providers in `models.yml`; a port mismatch is a rename or drift, not a config typo.
|